Absolutely amazing piece in the New York Times Magazine today by John Jeremiah Sullivan, about tracking down two female blues singers from the 1930s. Stop wasting your time on whatever it is you're currently skimming and read it right now.

I also very highly recommend Sullivan's book, Pulphead—a great read from start to finish. 

What happens when you hear a text rather than read it? The obvious thing is that you can do something else with your eyes. That is why I can listen to books when I garden. My hands and eyes can work. And so listening to a book is a different sensory experience than reading it. The inner imagining of the story becomes commingled with the outer senses — my hands on the trowel, the scent of tansy in the breeze.

—T.M. Luhrmann, Audiobooks and the Return of Storytelling—http://www.nytimes.com/2014/02/23/opinion/sunday/luhrmann-audiobooks-and-the-return-of-storytelling.html

I've always been curious about audiobooks but my primary activities—working (i.e. coding), reading, making music—require my full attention. When I'm really working, even regular conversation can be a huge distraction. And my longest commute is a 10-minute walk, which offers little opportunity for listening.

I wish I had a hobby that required my hands but not my conscious attention. I know I'm missing out on a lot—not just audiobooks but NPR and other podcasts that I hear about from friends. Theoretically, I could sit and listen attentively to NPR or a book, but somehow I never make the time. 

Maybe I'll try listening to This American Life next time I'm cooking a big meal.

My first startup job (at HOTorNOT) was essentially a "startup school." Though the site seemed frivolous on the surface, the team behind it was small, dedicated and fiercely data-driven–using homegrown tools—before being data-driven became a "lean" mantra.

These days, anyone can collect data simply by dropping in tracking code from Mixpanel, Heap, or even Segment.io, a meta-analytics service. But all this data collection is pointless unless you can use it to make decisions.

Collect correctly

The first requirement for data-driven decision-making is good data. Collect it all and make sure it's correct, that you're measuring what you intend to measure. It's easy to make mistakes here—whether it's a database query that's slightly off, or missing a segment in Google Analytics.

If you and your colleagues don't have faith in your data, then you won't be able to make decisions with it.

Ask good questions

After you've got good data, you should ask good questions. The more specific your question, the better—not "how many users have signed up", but "Which channels have high-value users come from in the last 6 months?" 

This question is slightly more complex than it sounds—you have to define what "high-value" means (and different people in your company might have different opinions)—but it's certainly answerable, and will give you more actionable analysis than just asking about "all users."

Analyze and Act

Next, analyze your data to answer your question. There are plenty of tools available for this—segmentation, funnel analysis, cohort analysis, plain old number-crunching in Excel. I won't go into detail here. But you should be able to answer the question you've asked if you've collected correct data.

The last step can be the hardest—acting on your analysis. If you're like most product designers, you'll often find data that disprove strongly-held opinions about your product. You may have to convince your team that your analysis is correct.

There will always be situations when you don't have enough time to rigorously collect and analyze data, or when you're creating something entirely new--these are times to use your intuition and make predictions based on previous experiments. But if you have the time, making an effort to drive your decision-making based on data will pay off.

Here's another easy win—never make users log in to unsubscribe from email notifications. For nearly every website, there's no security risk in allowing users to unsubscribe without logging in, and there are many benefits, including goodwill from users and fewer spam reports. Unfortunately, most popular sites don't take the simplest approach to solving this problem.

Bad approach: Twitter & Facebook

Twitter sends a lot of different kinds of notifications, and their approach to notification management is complex—as evidenced by their incredibly complex footer text, which wouldn't fit into 2 tweets, let alone one.

When you get a specific kind of notification (e.g. "@dzohrob is now following you on Twitter!"), you can disable *only* that notification type with one click. However, any other action requires you to log in, which means trying to remember your password and potentially going through a reset password process.

Let's say you remember your password—now you're presented with a list of 21 different notification types, some of which have sub-options. And of course, there's no way to unsubscribe from all of them. What an awful user experience.

Facebook takes a similar approach, allowing you to opt-out of one notification type, but requiring a login for any other management. Their notification management page is slightly better than Twitter's, but there's still no way to unsubscribe from all notifications with one click.

Good approach: Yelp

Yelp does it right. Even though they send many different types of notifications, like Facebook and Twitter, their unsubscribe link lets me manage the email I receive from them, and even lets me unsubscribe from all emails with one click. Nice work.

Bad approach: Feedburner

I'm not sure what kind of problem the "unsubscribe confirmation" page actually solves. Accidental clicks on unsubscribe links? Forwarded emails? Either way, it makes users do extra work for little benefit. I hope this pattern disappears over time.

Good approach: Campaign Monitor & MailChimp

Thankfully, companies like MailChimp that specialize in email delivery offer much better subscription management links. For example, I just got an email from SXSW—clicking through results in an instant unsubscribe, with the option to resubscribe.

Doing it the right way is easy

It's easy to choose a UI pattern for your unsubscribe links.

If your service sends many different types of email, follow Yelp's lead -- let users click through and manage all notification types without logging in. If you're sending a newsletter, let them unsubscribe with one click.

Make the right choice. You'll make your users happier and reduce your spam reports.

We've been making interactive web applications for since at least 1995, when Netscape shipped JavaScript (née LiveScript). Yet I still see the same basic usability mistakes on many websites.

I've started a list of the ones I think we should all agree on—a set of UI Rules, if you will. First on the list is one of the easiest to fix.

Rule #1. Focus on the first text field.

Most important for login and signup pages, but this rule applies to almost any form on any page (with the possible exception of pages with lots of separate forms). Why not save your users a click when they're performing the primary action on a page?

Let's look at some examples. I don't mean to pick on any of these companies—my methodology here was picking randomly off the list of trending startups on AngelList.

Swiftype fails in the basic case—there's a signup form on the front page, but the text isn't focused. The same is true of their login page.

Assembly does a better job—when you click "Login" on their page, the email address box has focus. They fail in a more subtle way, however—when I click "Signup," it takes another click to set focus on the Username field. (Perhaps they expect me to click on the Facebook signup option).

This is a common pitfall—when developers get deep into modal Javascript UI, details like input can get overlooked. Just to illustrate the complexity: if you visit their signup page directly, focus is set—and focus is kept when switching to the Login tab—but it's lost again when switching to the Signup tab.

The Fix

Developers often use JavaScript to solve this problem, but there's an even easier solution—the autofocus attribute on input fields. It's supported in all modern desktop browsers, but not on mobile (potentially for usability reasons according to this explanation from Wufoo).

Let's all agree to follow this rule from now on—however you do it, focus on the first text field on the page. In almost every case, you're saving your users time, which will lead to increased engagement and conversion rates. Together, we can make the Web a less frustrating place.

I left my old Kindle Keyboard 3G at a friend's house in San Francisco a few weeks ago, and used the opportunity to upgrade to the new Kindle Paperwhite. Amazon was kind (or crazy) enough to offer $3.99 next-day shipping on a Sunday night at 10pm—a deal I couldn't resist.

Unboxing a brand-new Kindle illuminates quite a bit of Amazon's product philosophy.

No retail packaging

The Kindle ships in a plain blue cardboard box, with little decoration. The only hint on the outside is a subtly embossed Kindle logo. Inside the box are the Kindle itself, a USB cable, and a quickstart guide. Minimal packagaing at its minimal-est.

This all makes sense because the Kindle isn't meant to be sold on store shelves. Amazon is the primary retailer, and consumers know what they're getting—therefore the box doesn't need to sell the product. Contrast this with Apple's box for Macbook Pros—even though they've toned it down in recent years, there's still the air of luxury about the packaging.

First-use experience

After turning on the Kindle (the battery has 50% charge—a nice touch), I can see where Amazon invested their time.

The tutorial is fantastic. It feels like Amazon took their cues from social games, where the on-boarding experience walks you through the major features step-by-step. Here, we learn how to turn pages, use the toolbar to change text size and backlight levels, and a few other common tasks.

And at the final step, there's even an upsell to Amazon Prime. Kudos to whomever put that step in there.

The end of physical books?

The new Kindle feels impressively responsive, and it's a great upgrade for owners of older Kindles. 

Of course, I have a few quibbles. It's not always clear whether I've successfully turned the page backwards, because the tap target for going back feels too small and there's no visual indication (other than the "location number" in the lower-left corner) of whether I've gone forward or backward. Also, having dedicated page-turning buttons on the Kindle Keyboard made it easier to read with one hand. Still, I'm confident in Amazon's lock on dedicated reading devices. 

As a lover of physical books, I know I'll still buy hardcovers and paperbacks -- they're one of the only physical items I still collect. But with a product this good, I can see why e-books are going to take over, and physical books will become the LPs of reading.

Reposting from the Lookmark blog, with apologies to the family member whose Facebook update I single out in this post. If you want to skip ahead, just sign up for Lookmark and tell me what you think.]

When I first joined Facebook, I updated my status and added photos often. It was a place for me and my actual friends to have actual conver­sa­tion­s—a welcome change from Myspace, which had been filled with animated GIFs and lots of quote-unquote “friends.”

Five years later, I barely update my Facebook at all—­maybe once or twice a month. When this update from a family member appeared in my News Feed a few months ago, I knew my relation­ship to Facebook had changed permanently:

Facebook has achieved their goal of building the best social graph on the Web. And that’s why I stopped updat­ing—every single update is shared to 700 current friends, former friends, random people from high school, coworkers and former cowork­ers, profes­sional contacts, immediate family, not-so-immediate-family, and randoms. Each update now forms a piece of our Timeline, to chron­icle all of our life events for posterity.

I stopped sharing because I don’t want every link I share to be perma­nently archived and viewed by any of my friends and family, or the HR depart­ment for my next job, or my future kids (if Facebook is still relevant by the time they’re old enough to use their iPhone 19).

This is exactly the problem that startups like Path are trying to solve. Path is meant to be a new, virgin terri­to­ry—a place for your real friends and family. It’s also the reason that I will never use Facebook Social Reader apps, like the Washington Post app, which automat­i­cally share all articles you read to your Timeline. I don’t want to share every article with my 700 friends, and I don’t want to perma­nently, publicly archive it on my Timeline. And I don’t want to see every article my 700 friends read, either.

Automatic sharing can be very compelling, though. When Last.fm launched (way back in 2002!) it was one of the first services to take advan­tage of automatic sharing—by simply installing the Audio­Scrob­bler plugin, you could share every­thing you played in iTunes, automatically.

While brain­storming ideas for new apps, I thought: “What about a Last.fm for news, a browser plugin that shares what you read automat­i­cal­ly?” I built a proto­type of the concept in a few days back in April to test the concept. When I showed it to my cofounders, we all agreed that it was very compelling, even with a total of 3 people using it—so we’ve spent the last few months building and refining the product. We call it Lookmark.

Lookmark is a Chrome exten­sion that automat­i­cally and privately shares what you’re reading from sites you choose (like the New York Times or links from Hacker News) with the friends you choose. Lookmark solves many of Facebook’s sharing problems because we designed from the ground up it for private sharing with a small group of close friends. It’s become an addic­tive source of news and conver­sa­tion over the past few month­s—and we’re just getting started.

We’re slowly opening Lookmark to new users. Inter­ested? Sign up at Lookmark.com and add me as a friend. I’d love to hear your feedback.

I’ll admit it—I love program­ming, and I have since I was a kid. This makes me a nerd, but at least I’m an honest nerd.

Code is a unique medium. There’s no activity better suited for the achievement of flow, which Wikipedia defines as “the mental state of opera­tion in which a person in an activity is fully immersed in a feeling of energized focus, full involve­ment, and success in the process of the activity.”

There’s something really special about hacking away on something and then, after hours or days of strug­gle—­making it actually work. And that joy of comple­tion, of having heroically strug­gled and then solved the problem, happens again and again. A good programmer working on a hard problem has little dopamine bursts firing in their brain all day.

I’m still not over it. And I think many people would benefit from (and enjoy!) learning how to code.

A computer on every desk, and in every home

Steve Jobs gets most of the credit for being a revolu­tionary technology leader, but Bill Gates was equally, if not more, vision­ary. In 1980, Gates defined the mission of Microsoft “A computer on every desk, and in every home.”1 It’s diffi­cult, now that we we all have incred­ibly powerful computers in our homes, cars, and pockets, to remember how insane this was at the time.

Microsoft executed that insane vision nearly flawlessly.

Computers became integral to many jobs in the 1980s and especially the 1990s, as the Internet penetrated workplaces and Microsoft’s Office software became ubiqui­tous. New software dramat­i­cally changed the nature of office work.

We’re about to experi­ence another dramatic change. The New York Times recently published The Age of Big Data. You should read it, but let me summa­rize: As computers become part of more things (phones, cars, toast­ers), they will generate more and more data, and we’ll need smart analysts to make sense of it all. Office workers will experi­ence a big shift like they did in the 90’s,as the ability to analyze data becomes more integral to every job—from manage­ment, to market­ing, to design, to front-line information workers.

I believe that simple computer profi­ciency will no longer suffice as software takes over more aspects of work and life. Not that it’ll be neces­sary for all office workers to become masterful program­mers; but those who can write simple programs to help automate their daily work and make sense of piles of data will have an advantage.

This change means more people will have the oppor­tu­nity to experi­ence the joy of code. Unfor­tu­nately, learning requires hard work, and most people haven’t been educated in the basics of computational thinking.

How will all these people learn how to code?

Learn BASIC now

When I was around eight years old I asked my parents to buy me a book called Learn BASIC Now. (Amazingly enough, Amazon still sells the book for a penny.) I remember sitting down at the computer in our living room and working through the book’s exercises in Quick­BA­SIC, a program­ming language included with most PCs at the time.

Within a few weeks I was able to make simple video games, and by the time I was in middle school I was asking my parents for related stuff—­more books, then an Internet connec­tion, then a domain name (which cost $100 at the time!) and Web hosting service, so I could help others learn.2

One of the most popular games I wrote as a kid had primi­tive graphics that looked like this:

This wasn’t state-of-the-art for 1994, but it was playable and maybe even fun. I called the game Lander. The goal was to safely pilot your ship to a landing strip on the moon, avoiding obsta­cles along the way3. I really enjoyed making games and sharing them with friends and on the Web—it was a great motivator for me to improve as a programmer.

But what worked for me would­n’t work today. The everyday systems we use have become more compli­cated, and it’s harder to create a simple program of acceptable quality.

Most DOS users would be able to find and run GW-BASIC or Quick­BA­SIC, two program­ming languages that shipped with the operating system—but although current Macs ship with several powerful languages installed, most newcomers would­n’t know where to find them. If they could, the Terminal would seem scary because the Mac user inter­face covers up its text-based underpinnings.

Even then, creating a text-based program would­n’t seem that impres­sive—the simplest iPhone apps come with beauti­ful, animated inter­faces that react instantly to our touch, courtesy of Apple’s design tools. A text-only number guessing game seems pretty lame by comparison.

Learning to code in 2012

So how does one get started, given the complexity of today’s computers and the high expec­ta­tions of users? Thank­fully, many startups and univer­si­ties have begun tackling the problem in innovative ways.

Udemy offers a class by Zed Shaw called Learn Python the Hard Way for $29. Zed also offers a free HTML book for the course. Zed’s premise is that the best way to learn is by doing, so you’ll be doing a lot of typing. This seems to me like a Good Thing.

Another startup called Codecademy offers a unique way to learn—sim­ple, step-by-step tutorials that run in your web browser, allowing you to inter­ac­tively learn how to code without installing any software or buying any books. They’ve recently received a lot of press for a 52-week program called Code Year. When you sign up for Code Year, you receive a weekly code lesson via email which you can complete on their website. There are also Q&A forums where you can get help with each lesson. Hundreds of thousands of people have signed up, including high-profile folks like Mike Bloomberg.

There are a couple of poten­tial downsides to Code Year and Codecad­emy. Currently, they only teach Javascript, which I would­n’t recom­mend as a first language for a begin­ning program­mer. Second, the inter­ac­tivity of the lessons hides some of the reali­ties of coding. Program­ming is a unique activity because for the most part it’s a very solitary journey—y­ou’re alone with your work, shuffling logic around your brain. I’m curious if the folks who go through Codecademy tutorials are ready for the kinds of frustra­tions they’ll face—Learn Python The Hard Way is much more like “actual” programming.

There are tons of other great resources, far too many to list here. The Khan Academy has great video lectures; Google offers excellent Python tutorials; MIT offers their Intro to Computer Science and Programming, and Stanford offers an Intro to Computer Science and Programming Methodology..

Regard­less of which method you end up trying, if you want to learn, find something that appeals to you and stick with it. Don’t get discour­aged; the rewards will reveal themselves over time.

Conclusion

Learning to code is like learning to play an instru­ment. Some people will achieve basic easier than others. But for nearly every­one, becoming a good programmer will take many hours of delib­erate practice; it’s not something you can achieve in hours or days, no matter how many tutorials you read.

To be a great program­mer, you have to really want to learn; it’s not going to happen in a day or a week. Thank­fully, you probably don’t need to become a great programmer unless you plan to pursue a career in software engineer­ing. You can learn enough code in a short time to improve and automate some of your daily work.

I hope that the changes ahead as we enter the “Age of Big Data” give more people the oppor­tu­nity to experi­ence the joy of code.

I’d love to hear from people who want to learn code. If you find a particular resource linked in this post helpful (or terrible), or you think I’ve missed something, please let me know.

It’s been almost five years since I started my last startup.

I remember clearly the heady blend of optimism and anxiety that came with filing our incor­po­ra­tion papers, followed by the pride and sweat and joy of brain­storm­ing, building and launching our first real product.

I just started a new startup with two great friends, Amit and Drew, and those feelings are back. The five inter­vening years have brought successes, failures, and hard-learned lessons, yet I could­n’t feel more sanguine about the future.

I should probably be more worried about the months ahead­—there’s nothing to anchor me and my cofounders except our own deter­mi­na­tion and wits. It’s a long, hard road, and startups fail for a lot of reasons.

Yet despite all that, I still feel optimistic.

I’ll do my best to correct my mistakes from my last go-round1. Yet even if I fail this time, I feel like I’ve already won. Inertia is one of the strongest forces in the world—there are so many different things acting to prevent one from doing something, from making something. And somehow, we’ve managed to give ourselves the oppor­tu­nity to really create something awesome.

All we have to do now is make the most of it.

Paul Graham summa­rized a core tenet of internet entre­pre­neur­ship simply: “You make what you measure.” 1

There’s something magical about putting a number somewhere you can see it every day, whether that number repre­sents daily profit, new customers, or something else. Suddenly one wants to improve, especially when the metric is paired with a pretty graph showing one’s improve­ment over time. It becomes a game, and everyone loves a good game.

I’ve employed this tactic at nearly every company I’ve started or worked at, measuring as much as I can—then choosing key metrics to create a company-wide dashboard that I could look at every day. It’s remark­ably simple and effective.

I got sick a few weeks ago and spent too much time in bed, lazily wandering the Inter­net, where I ran across a compelling video about the benefits of 30 minutes of activity a day. The kicker, if you don’t have a few minutes to watch it: 30 minutes a day of pretty much any exercise—walking includ­ed—has been shown to offer dramatic health improve­ments, in mood, sleep, heart health, and more.

I was convinced, and wanted to take action. A computer program­mer’s life is neces­sarily seden­tary, and I wanted to counteract the negative effects of too much inactiv­ity. So I bought something that had been on my wish list for awhile, the Fitbit Ultra.

Fitbit is a small, wearable device that provides dead-simple measure­ments of your physical activ­ity. It measures the number of steps you’ve taken, floors climbed, distance traveled, and calories burned. Fitbit also functions as a basic sleep track­er—when worn on your wrist at night, it measures general sleep patterns, including the number of times you awoke during the night.

After carrying the Fitbit around in my pocket for two weeks, I can uncon­di­tion­ally recom­mend it to anyone looking to improve their health. It’s been fasci­nating to see my physical activity throughout the day—

and my patterns of sleep at night—

Counting the flights of stairs I’ve climbed each day has already encour­aged me to skip the elevator more often at home. I often find myself taking longer routes while walking, or walking instead of taking the subway.

It really works.

That said, the device isn’t perfect. Right now it must have an optimistic idea of my stride, because its distance measures (and hence calories burned) are off by a fair amount. I’ve been meaning to measure my stride accurately but it’s diffi­cult to do when you live in a tiny apartment.

Also, while the Fitbit can measure exercise like running and ellip­tical machines easily, it can’t measure activity that won’t trigger its pedome­ter, like swimming or using a stationary bike. Fortu­nately, the website lets you log your exercise and can compute calories burned for you.

It’s too early to say whether I’ll be as enamored with the Fitbit in a few months as I am right now, but so far the entre­pre­neur’s maxim has held true: you make what you measure.

If you’re trying to live health­ier, try getting a Fitbit.

1 Source: Startups in 13 Sentences

When I started my last startup, I kept quiet about my work. I didn’t tell anyone what I was working on, didn’t blog or tweet or promote myself or my apps.

I put my career in stealth mode.

At the time, I had several reasons to stay quiet. I wanted the freedom to explore multiple projects without the pressure of commit­ting fully to one of them. I hated startup­s’ “About Me” pages, full of smiling people bragging about themselves in the third person. And if my startup failed, no one would know—I could quietly move on to the next project.

Five years later, I see this as a huge mistake, maybe the biggest one I’ve made in my career.

Because I have almost no public presence on the Inter­net, you probably have no idea that I’m a programmer (let alone a good one) unless you know me person­ally. Yet I spent most of my time from 2007 through 2011 writing thousands of lines of code, powering apps used by millions of people.

Now, most of that code is dead. Thousands of hours of my life, literal years of work—gone.

My cofounder and I wrote some pretty cool libraries, including:

• an iPhone table view markup language we called JSTN (JavaScript Table Notation, or Justin, a play on JSON, the JavaScript Object Notation)
• a simple command language to allow a server to dynam­i­cally control iPhone app flow, such as pushing a new ViewCon­troller, popping up an alert, etc (called SICC, pronounced sick, for Server Invoked Client Commands)
• many Rails hacks to allow seamless caching, distrib­uted reads and writes, and database sharding

Some of this code lives on in the apps that my cofounder still runs, but nobody knows about it. Like most of my work from the past five years, it’ll probably languish and eventu­ally die. But If we had open-sourced any of these libraries, I’d wager that at least one (and possibly more) of them would still be living today, because they solve real problems.

I recently left my job to start a new startup. One of my goals is to open-source as much work as I can, because I want my code to be used and improved by others. And I’ll write about what I’m doing, so I can share my successes and my failures.

I don’t want to repeat my past mistakes. I’m going to make my work visible, for better or worse. So I’m taking my career out of stealth mode.

There’s some discussion on Hacker News as well.

Writing about loss is diffi­cult. Few can success­fully describe the empti­ness one feels after the sudden death of a parent, child, friend.

And yet we try. When my mother passed away nearly twelve years ago, I started a web journal. Writing helped me process my thoughts and feelings into something concrete and more understandable.

Joan Didion described loss with quietly powerful language in The Year of Magical Thinking and Blue Nights, her memoirs about the death, in shock­ingly short succes­sion, of her husband and daugh­ter. Despite the painful subject matter, I highly recom­mend reading them (in order), both to people who have experi­enced a sudden loss and to those who haven’t. Didion captures the empti­ness of life without her husband so well—the immediate eleva­tion of once-unimportant miscel­lany to artifacts, the new strange­ness of once-familiar daily routines.

William Maxwell’s So Long, See You Tomorrow is a short, moving work about memory, family, and loss, set in a small town in Indiana. Like The Sense of an Ending, most of the action occurs in the distant past, as an elderly narrator looks back on his life. And, like Sense, So Long can be read in an evening, though I’d recom­mend allowing some time for the words to sit in your mind. Maxwell uses decep­tively simple language throughout the book; it took me some time to realize how carefully he chooses his words, and their cumula­tive effect is powerful.

I’m not sure how long a quote is too long, but I found this passage near the begin­ning of the book exceptional:

My father was all but undone by my mother’s death. In the evening after supper he walked the floor and I walked with him, with my arm around his waist. I was ten years old… Because he didn’t say anything, I didn’t either… His eyes were focused on things not in those rooms, and his face was the color of ashes…

I had to guess what my older brother was think­ing. It was not something he cared to share with me… At night we undressed and got into bed and fell asleep without taking advan­tage of the dark to unburden our hearts to each other. It strikes me as strange now. It didn’t then… What I didn’t say, across the few feet that separated our two beds, was that I could­n’t under­stand how it had happened to us. It seemed like a mistake… I had to find an expla­na­tion other than the real one, which was that we were no more immune to misfor­tune than anybody else, and the idea that kept recur­ring to me, perhaps because of that pacing the floor with my father, was that I had inadver­tently walked through a door that I should­n’t have gone through and could­n’t get back to the place I hadn’t meant to leave. Actually, it was the other way round: I hadn’t gone anywhere and nothing was changed, so far as the roof over our heads was concerned, it was just that she was in the cemetery.

It’s not fair to describe So Long, See You Tomorrow as a book strictly about loss; the plot revolves around a murder, the details of which are revealed slowly. But all of the charac­ters feel loss of some form or anoth­er—death, betrayal, separation, regret.

It’s a beautiful book. You can probably find it in your local used-book store, and on Amazon.

Atul Gawande expanded a great piece in the New Yorker from 2007 into The Checklist Manifesto, a short, compelling read about how a simple tool can help humans tackle complexity.

Most of the book explores reducing human error in medicine (spoiler: good check­lists prevent mistakes). But Gawande includes a fasci­nating chapter on the construc­tion industry where he writes:

For most of modern histo­ry… the dominant way people put up build­ings was by… hiring Master Builders who designed them, engineered them, and oversaw construc­tion from start to finish, portico to plumb­ing. Master Builders built Notre Dame, St. Peter’s Basil­ica, and the United States Capitol build­ing. But by the middle of the twentieth century the Master Builders were dead and gone. The variety and sophis­ti­ca­tion of advance­ments in every stage of the construc­tion process had overwhelmed the abili­ties of any individual to master them.

The solution? A combi­na­tion of delega­tion, commu­ni­ca­tion, and checklists.

I see many paral­lels to the computing indus­try. Thirty years ago, Steve Wozniak designed and built a complete computer by hand, the now-legendary Apple I. Today, I carry a computer many orders of magni­tude more complex than the Apple I in my pocket: the iPhone. There’s no one person who can under­stand every single compo­nent of the iPhone and how they work together, let alone assemble one from scratch.

Our solution in both hardware and software is abstrac­tion. Different special­ists and compa­nies handle different compo­nents, and agree on inter­faces between those compo­nents, much like the plumbers and electri­cians and HVAC special­ists collab­o­rate on a skyscraper.

In software, abstrac­tions have evolved at the same speed as the under­lying complex­ity, so that appli­ca­tions like the webserver Nginx and the game Minecraft can still be largely written by one person. The tools just keep improving.

The Apple iPhone SDK is a great example of an excep­tional tool: for (almost) no cost, Apple offers devel­opers a complete coding environ­ment, UI compo­nents to make beautiful inter­faces, a distri­b­u­tion channel, and payment process­ing. It’s never been easier for a team of one or two people to make great software that reaches a large audience.

Moreover, program­mers have created many varieties of automated check­lists to fight against increased complex­ity, from unit tests to full functional tests that click on buttons within an app.

Without such tools the age of the programmer “master builder” would have ended long ago. I’m curious if we’ll ever reach a point where one or two people can no longer create meaning­fully useful applications.

The Checklist Manifesto can be found at Amazon and elsewhere.

Several years ago I decided to start collecting jokes.

I was inspired by this 2009 New York Times article quoting Robert Provine, a professor of psychology at the Univer­sity of Maryland, explaining that “[w]hat makes a joke successful are the same proper­ties that can make it diffi­cult to remem­ber.” Jokes are hard to remember because memory relies heavily on pattern-matching, and humor often arises directly out of breaks in our usual patterns.

So I made a conscious decision to remember jokes, much in the same way I made the decision to actively remember people’s names after reading Dale Carnegie. They’re both great party tricks.

Most of my close friends know my favorite jokes by heart; I habit­u­ally recite them in awkward conver­sa­tional pauses, at parties, in bed. This one has been in my reper­toire for years:

Did you hear about the two TV antennas that got married? The ceremony was boring, but the recep­tion was great.

I still think it’s funny, even if my girlfriend doesn’t.

When my friends could­n’t supply me with enough new jokes to annoy other friends with at parties, I turned to my Internet pal Google for a little assistance.

Google can answer almost any question that comes up in conver­sa­tion—who invented the zero, for example (long story, but big ups to al-Khwārizmī1), or when Paul Simon’s album Songs from the Capeman was released (1997, not that you asked)2. A search for “good jokes”, in contrast, turns upFuNNy JoKeS make life gOoD and HuMoRouS in the top spot, which gives us this gem:

We had grandma for Christmas dinner ?
Really, we had turkey !

This might be the most brilliant anti-joke ever written. Or maybe something was lost in translation.

Okay, the first result isn’t always the best. Let’s move on to #2, danggoodjokes.com. Each joke linked from its nearly illeg­ible homepage contains one or more jokes along with some (usually conser­v­a­tive) commen­tary on just how gosh-darn crazy our world has become, complete with rainbow WordArt headlines:

Even when they are readable, none of the jokes are funny.

The third Google result, smileJoke, brings some modern “so­cial web” flavor into the mix. smile­Joke allows users to submit ratings for each joke, and appar­ently, all the jokes have a rating of 1. This is not surprising given the material:

My wife had gained a few pounds. It was most notice­able when she squeezed into a pair of her old blue slacks. Wondering if the added weight was notice­able to everyone else, she asked me, “Honey, do these slacks make me look like the side of the house­?” “No, dear, not at all,” I replied. “Our house isn’t blue.”

Friends, these are the top 3 sources for “good jokes” on Google. I guess Google’s ranking algorithm doesn’t have a sense of humor. But the next time your girlfriend asks you whether her jeans make her look like “the side of the house­,” you now know how to get the laughs.

The best source I’ve found is, surprisingly, the Jokes subreddit. It suffers from the usual Reddit demon­s—trolls, memes and misogyny abound­—but some of the jokes are funny, which puts it in the top 0.001%. Recently, Reddit ranked this gem as the top joke of the day (revised for clarity):

Q: What do free health­care and good jokes have in common?
A: Ameri­cans don’t get them.

Not bad. Still, the Internet disap­points when it comes to humor.

Why is it so hard to find good jokes? Humor is subjec­tive, of course, but is it more subjec­tive than, say, music? Google and others have used machine learning to solve problems that once seemed insol­uble, to find patterns and meaning in vast troves of data.

How come no one has applied this technology to finding jokes that are actually funny?

After living in New York for a year and a half, I miss San Francis­co’s excel­lent used bookstores. I feel lucky to work just half a block from The Strand, but there’s something about a small bookstore’s curated selec­tion of used books that makes browsing more fun.

I was working in SF last week when I happened into Aardvark Books one night after dinner. Aardvark boasts a great “re­cent arrival­s” section, just a few shelves long but brimming with excel­lent, inexpen­sive reads. I randomly picked up a hardcover copy of John McPhee’s Uncommon Carriers, based mostly on its excel­lent cover art and a vague recol­lec­tion of having read some of his work in theNew Yorker. (It helped that some bookstore employee wrote in pencil on the first page, “Pretty good!”)

It was a lucky find. Uncommon Carriers explores freight trans­port and the people who work Ameri­ca’s trucks, barges, ships, planes, and trains, day and night. This might sound boring at first glance, but McPhee’s writing is strangely hypnotic and enthralling.

One piece in particular, Out in the Sort, kept me up late last night. The piece begins with a detailed descrip­tion of a lobster-catching opera­tion in Newfound­land, and uses the lobster compa­ny’s trans­port needs as a segue into a decon­struc­tion of how UPS operates. Describing their world­wide hub in Louisville, Kentucky, McPhee writes:

This labyrinth, which outthinks the people who employ it, is something like the interior of the computers that run it. Like printed circuitry, seven great loops, each a thousand feet around, are super­posed at right angles above other loops… Unending sequences of letters and small packages zip around these loops, while the larger packages follow one another on the belts, each package tailgating the one in front of it but electron­i­cally forbidden to touch it… Collec­tively, the loops are like the circuits in the mother­boards among the inter­face cards of a central processing unit wherein whole packages seeking specific airplanes are ones and zeroes moving through the chips.

There’s too much to glean from the piece (and from the book) for a short blog post, but I’ll pick a few tidbits. We learn that UPS employs college students to deal with “ex­cep­tion­s”—­pack­ages that require human inter­ven­tion. When the company experi­enced diffi­culty hiring enough students, it founded and endowed a college of its own, Metropolitan College, in concert with the Univer­sity of Louisville and other local schools. UPS employees also service Toshiba laptops on-site, and keep the largest supply of Bentley parts in the world. McPhee’s great writing and sense of humor keep everything interesting.

There are hundreds of other fasci­nating details. If you have a New Yorker subscrip­tion, you can read Out in the Sort on newyorker.com. If not, I managed to find a copy online, which you can download from Mediafire while it lasts. Either way, if this sounds at all inter­esting to you, you should find a copy of Uncommon Carriers. I know I’ll be looking for more McPhee titles next time I hit a used bookstore.

…when you are young, you think you can predict the likely pains and bleak­nesses that age might bring. You imagine yourself being lonely, divorced, widowed; children growing away from you, friends dying. You imagine the loss of status, the loss of desire—and desir­abil­i­ty… What you fail to do is look ahead, and then imagine yourself looking back from that future point. Learning the new emotions that time brings. Discov­er­ing, for example, that as the witnesses to your life dimin­ish, there is less corrob­o­ra­tion, and there­fore less certainty, as to what you are or have been. Even if you have assid­u­ously kept record­s—in words, sound, pictures—you may find that you have attended to the wrong kind of record-keeping.

Sense of an Ending is a short and wonderful book, a slim 180 pages readable in a single sitting. There’s much to savor, though, and like Marilynne Robinson or Alice Munro, Barnes chooses every word carefully and precisely. Highly recommended. Amazon

New York City lacks the startup density of Silicon Valley, but there are plenty of reasons to start your next company here:

1. A great (and growing) startup scene. Soho, Union Square and the Flatiron district form the new Silicon Alley, where you’ll have your pick of compa­nies, meetups and events. Great startups are doing great work here across industries—including tumblr, foursquare, 10gen, Betaworks, turntable.fm, Etsy,Skillshare, and Harvest, among others 1. New York also has a sizable angel and VC community 2.

2. Bubble-free. SF often feels like a company town; in contrast, New York is a national hub for publish­ing, adver­tis­ing, fashion, food, and finance, to name a few. Being in the center of all these indus­tries is inspir­ing, and your startup can take advan­tage of this diversity.

3. The singles scene. Some folks might rank this as #1; I’m trying not to assume too much. But if you’re a single straight guy or gal in SF or Mountain View, you (or a friend) have likely complained about the singles scene in the Bay Area.

Forbes ranks New York as the best city for singles, putting it above San Francisco, which clocks in at #7 3. For single guys, the numbers don’t lie—there are 210,820 more single women than men in the New York metropol­itan area, while the Bay Area boasts a surplus of 65,000 men. 4 



Plus, being a hacker/en­tre­pre­neur is more unique in New York—telling someone you make iPhone apps elicits inter­est, not eye-rolls. (Try it.)


4. Official support. Mayor Bloomberg wants to turn New York into a tech hub, and city officials reach out to local entrepreneurs. The NYC Economic Development Corporation offers great resources including info on city incen­tives and discounted office space for startups.

5. The city never sleeps. NYC is much friendlier to the late-night hacker’s sched­ule—­sub­ways and buses run 24/7 and great restau­rants stay open till 2am or 4am. Contrast this with San Francisco, where it’s hard to get good food after 11pm, even on a Saturday night.

6. The upstart startup hub. This one is hard to quantify, but there’s something exciting about being part of the NY startup culture because it is less estab­lished than SF—y­ou’ll find plenty of room (and enthu­si­asm) for starting a new meetup, for example. There’s an energy in the air here, a sense that we are all part of something new and growing and awesome.

If you’re living in SF or elsewhere and looking for a change, come visit NYC and see if it’s right for you.

I’ll even buy you a coffee.

Notes

1 See a bigger list of NY-based companies here: Internet Made in New York City.

2 The New York market on AngelList hosts an excel­lent list of NYC-based investors and angels.

3 Source: Forbes

4 Source: Boston Globe

“The tendency to see patterns in randomness is overwhelming…”

Daniel Kahneman’s Thinking, Fast and Slow made many “best of 2011” lists this year, and deservedly so. Plenty of books on the system­atic faults of the human mind have made the rounds, from Ariely’sPredictably Irrational and its followup to Nassim Taleb’s Fooled by Randomness and The Black Swan.

Fooled by Randomness was a great book marred by the pompous­ness of its author. In fewer words (and with much less ego), Kahneman covers similar terri­tory in more depth and backed by better research:

How many good years should you wait before concluding that an invest­ment adviser is unusu­ally skilled? How many successful acqui­si­tions should be needed for a board of direc­tors to belive that the CEO has extra­or­di­nary flair for such deals? The simple answer to these questions is that if you follow your intuition, you will more often than not err by misclas­si­fying a random event as system­atic. We are far too willing to reject the belief that much of what we see in life is random.

I will do my best to remember the above quote whenever I jump to conclu­sions about a startup, an entre­pre­neur, even an A/B test.

There’s too much in the book that’s applicable to anyone starting a business, or inter­preting analytics from your website, or trying to make rational decisions. Instead of quoting from every chapter, why not just check it out for yourself? (No affil­iate link, I promise.)

Steven Levy did a great interview in Wired of Amazon CEO Jeff Bezos. The money quote, for me:

If every­thing you do needs to work on a three-year time horizon, then you’re competing against a lot of people. But if you’re willing to invest on a seven-year time horizon, you’re now competing against a fraction of those people, because very few compa­nies are willing to do that… At Amazon… [w]e’re willing to plant seeds, let them grow—and we’re very stubborn. We say we’re stubborn on vision and flexible on details.

I know I’ve person­ally been too focussed on the short-term in the past—and any casual browser of the new startups on Angel­List will see plenty of fast-follow startups with little to differ­en­tiate them from the competition.

Bezos is a great CEO, and doesn’t get the credit he deserves. Amazon is an amazing company.

It is much easier to start a web startup in 2011 than it was in 1995, or even 2005.

You don’t need upfront capital to buy racks of machi­nes, miles of CAT-5, routers and load balancers, redun­dant power and backbone connec­tivity — all are avail­able on-demand from Amazon for pennies an hour.

You don’t need to write server-side Web frame­works — you have your pick in nearly any language. In fact, you have your pick for any number of services you might once have had to develop or maintain yourself: email deliv­ery, web analyt­ics, A/B testing, server monitor­ing, file hosting.

Because the infrastruc­ture cost of starting a startup has gone down, the amount of money you need from investors to get your company off the ground has also decreased. You can build a proto­type in a few months (as every YC class demon­strates, twice a year) for less than $20,000.

And there are more people to give you that money — VCs, ex-Googlers and Facebook­ers, other angels, startup accel­er­a­tors and incubators.

But ask any startup around what they really do need right now, and the answer is clear — great technical talent. Startups of all sizes are competing for a small pool of highly skilled candi­dates, and in startup hubs like San Francisco and New York the compe­ti­tion for engineers is fierce. This scarcity has persisted despite consis­tently high national unemployment rates.

And because it has become so easy and cheap to start a startup, convincing great engineers to join an existing startup has gotten much harder.

All of these trends will continue — infrastruc­ture costs will continue to decrease, tools for creating web products will get more sophis­ti­cated, and more people (and more kinds of people) will be investing in startups at all stages.

But those startups who can hire and retain great talent will achieve greater success because they have the capacity to out-execute their peers.

Note: this is the third post in a series on email delivery.

After ignoring this series for a couple of months, Garry from Posterous submitted my original post toHacker News a few days ago. Since a few people seem to have found it useful, I’ve got a renewed motiva­tion for hammering out a few more posts.

All of the major ISPs and email providers have some propri­etary methods of inter­acting with large-volume email senders. Most include a feedback loop, which notifies email senders when a recip­ient marks a message as spam. This post will help you get set up with the major web-based email provider­s—Mi­crosoft, Yahoo!, AOL, and GMail.

Microsoft Hotmail & Live Mail

Hotmail is still one of the largest email services out there, even though if you live in the Valley you probably know zero people who use it. Because it is such a large attack surface, Microsoft­’s spam filter (called SmartScreen) is pretty aggres­sive, especially with mail coming from IP addresses with no sending history. Fortu­nately, they’ve imple­mented two major programs to help legit­i­mate email senders get into the inbox.

• Smart Network Data Services — SNDS gives you deliv­er­ability infor­ma­tion based on the IP of your mailserver. You’ll need a Hotmail account to use as your SNDS login. Fill out the access request form, and you’ll receive a verifi­ca­tion link in an email to abuse or postmaster@yourdomain. 
  
  SNDS requires a minimum volume of about 100 emails per day in order to give you any data. Once your IP is activated, you’ll get infor­ma­tion on how your emails are being treated by Hotmail, including data on the number of unknown addresses, complaint rates, spam trap hits, and more. See the SNDS FAQ for more detailed information.

• Junk Mail Reporting Program (JMRP) – This is Hotmail’s feedback loop program. Fill out thisMicrosoft Support questionnaire after signing up for SNDS above. There will be a legal document signing process that you’ll have to go through, but once you’re set up, you’ll receive an email to a speci­fied address for every spam complaint from a Hotmail user. You should remove complaining users from your sending lists in order to preserve your sender reputation.

The Postmaster Services home page has more infor­ma­tion on both programs.

Yahoo!

Yahoo! Mail is the largest email provider in the US. Unfor­tu­nately, their feedback loop program has been closed to new appli­ca­tions for a year or more—see this FAQ answer on the Yahoo! postmaster site for more details.

As of January 2009, your best bet is to first read Yahoo!’s sending best practices, make sure you comply, and then fill out the Bulk Sender Form. If you only send double-opt-in mail (ie, you’re not sending user-generated invita­tions), you may be eligible for their Whitelist as well.

If you happen to have any infor­ma­tion on the Yahoo! feedback loop process, please leave a comment!

AOL

Plenty of low-tech US users still use AOL mail. They make very clear their email sending best practices, much like the other providers. Sign up for the feedback loop here. If you only send double opt-in mail (again, no user-generated invites), you can sign up for their whitelist.

GMail

As befits Google’s philos­o­phy, there’s very little human input involved in their spam filter­ing. Here are the GMail bulk sending guidelines. If you do run into deliv­er­ability issues here, there is a support form to contact them as well.

I person­ally haven’t had many issues getting email deliv­ered to Google — as long as you set up SPF and DKIM correctly (see my previous post in this series) and keep your complaint rates low, your mail should go into the inbox.

Funny how quickly six weeks can pass without an update; I’ve been meaning to finish my series on e-mail deliv­ery, and maybe write more about some of the music and books I’ve been appreciating lately.

This passage from The Love of a Good Woman by Alice Munro caught me today:

She still says this every once in awhile.

“What I remember most is that I could­n’t touch you and wondering if you understood.”

Karin says yes. She under­stood. What she doesn’t bother to say is that back then she thought Rosemary’s sorrow was absurd. It was as if she was complaining about not being able to reach across a conti­nent. For that was what Karin had felt she had become—­some­thing immense and shimmering and suffi­cient, ridged up in pain in some places and flattened out, other­wise, into long dull distances. Away off at the edge of this was Rosemary, and Karin could reduce her, any time she liked, into a config­u­ra­tion of noisy black dots. And she herself—Kar­in—­could be stretched out like this and at the same time shrunk into the middle of her terri­tory, as tidy as a bead or a ladybug.

I find myself returning to Munro’s stories even though I’m in the middle of reading a couple of other books right now, both for their emotional immediacy and how well their length and struc­ture lends itself to my current schedule.

In college all of them had studied the putative effects of deraci­na­tion, which were angst and anomie, those dull horrors of the modern world. They had been examined on the subject, had rehearsed bleak and porten­tous philoso­phies in term papers, and they had done it with the earnest suspen­sion of doubt that afflicts the highly educa­ble. And then their return to the pays natal, where the same old willows swept the same ragged lawns, where the same old prairie arose and bloomed as negli­gence permit­ted. Home. What kinder place could there be on earth, and why did it seem to them all like exile? Oh, to be passing anony­mously through an imper­sonal landscape! Oh, not to know every stump and stone, not to remember how the fields of Queen Anne’s lace figured in the childish happi­ness they had offered to their father’s hopes, God bless him.

Passages like this make me think that Marilynne Robinson may be my new favorite author. Home is a wonderful book, beautiful and moving and written with such evoca­tive, precise language. I can’t recom­mend it enough. But if my recommen­da­tion hasn’t convinced you, please read James Wood’s excellent piece in the New Yorker about Robinson and her work.

Note: this is the second post in a series on email delivery.

So you’ve just set up your own email server and you’re ready to send your first message. You type in your address, click “Send,” and wait. And wait. Chances are, the message will end up in your spam folder, if it shows up at all.

There are a few existing proto­cols that will greatly improve your chances of getting into the inbox. They are, in increasing order of complexity:

• Reverse DNS
• SPF / Sender ID
• DomainKeys / DKIM

Reverse DNS

The easiest way you can improve your deliv­er­ability is to make sure that a reverse DNS lookup on your mail server returns the server’s hostname. Usually you have to ask your ISP to set this up.

Let’s look at a Digg mailserver for an example. From a UNIX prompt, you can type dig mail.digg.comto get this result (or something like it):

mail.digg.com. 1139 IN A 64.191.203.36

Now, if you check the reverse DNS record for that IP address with dig -x 209.191.118.103, you’ll see that the reverse DNS matches up:

36.203.191.64.in-addr.arpa. 3572 IN PTR mail.digg.com.

Reverse DNS is required for good deliv­ery. Without it, ISPs may reject the message. Hotmail might accept the mail but not deliver it (it just vanish­es), or just throw your mail into the spam folder.

SPF / Sender ID

Sender Policy Framework is a simple protocol for speci­fying which servers are allowed to send mail for a partic­ular domain. All you have to do is set up a TXT DNS record according to a simple format.

Let’s look at Reddit as an example. A quick dig reddit.com txt will pull up their TXT:

reddit.com. 212 IN TXT "v=spf1 mx ip4:208.96.53.70 mx:mail.reddit.com ~all"

Tearing the SPF record apart, we get:

• v=spf1 identi­fies an SPF specification
• mx allows the mail servers for reddit.com to send mail (those listed in its MX records)
• ip4:208.96.53.70 autho­rizes a single IP to send mail
• mx:mail.reddit.com allows any MX servers for mail.red­dit.com to send mail as well
• ~all speci­fies that any machi­nes/ad­dresses NOT listed here are not autho­rized to send mail. The~ indicates a “tran­si­tional mode” – once you’re done testing your record, use a dash (-all).

The specification is here, and this tool will help you set up your own record. If you’re sending invites or user-generated mail, check out this OpenSPF best practices page for infor­ma­tion on how to make your invita­tion show up as “From:” a partic­ular user, but ensure that any bounces come back to your mailserver.

A quick note on terminology: Microsoft has a technology that they call Sender ID, which performs the same function as Sender ID but with a slightly different format. For most practical purposes, a valid SPF record is a valid Sender ID record, and when a Microsoft service talks about Sender ID, just read it as SPF. For the curious, the Wikipedia article on Sender ID has more details.

DomainKeys and DKIM

Both DomainKeys and DKIM (DomainKeys Identi­fied Mail) are DNS-based proto­cols for email authen­ti­ca­tion using a public key speci­fied in your DNS record. Before sending a message, your mailserver “sign­s” the email and puts the result in a header on the message. Any recip­ient can verify that the message origi­nated from your servers by checking the public key against the signature header.

DomainKeys is an older standard created by Yahoo!, which is now being replaced by the DKIM standard. Because they are different standards, and different ISPs only support one or the other, you’ll need to set up both DomainKeys and DKIM signing.

Yahoo! and GMail both give a valid signa­ture high impor­tance in deter­mining whether or not a message is spam, and usually tell the user that a message is authen­ti­cated. For example, GMail shows a “signed-by” field:

Let’s take a peek at a record in the wild. Twitter’s DKIM public key can be found by runningdig default._domainkey.twitter.com txt, which returns:

default._domainkey.twitter.com. 600 IN TXT "g=\;" "k=rsa\;" "t=y\;" "p=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAN+FNJESkUBl+vuJDPsL3RSgYI9Qzlq43+l7Q72pRZRDprrhZTXIi7NdSqy+f9hn" "pet1pKMYMYnCxgmaS3qhUXMCAwEAAQ=="

The longest section of this record is the key itself, after the p=. Some registrars don’t let you store TXT records this long, and if you get an error setting up your public key, you might have to switch to a new DNS provider. You can learn about the other fields from the DKIM Spec.

If you’re using Postfix, DKIMProxy does both DomainKeys and DKIM well, and there are decent step-by-step instruc­tions its homepage. You can also check out the SourceForge DomainKeys page to find software for different MTAs as well as testing tools.

Next steps…

Reverse DNS, SPF, and DKIM should definitely help your deliv­er­ability rates – if you weren’t getting through at all, you should at least be hitting the Spam folder now. However, there’s more work ahead. Most ISPs have programs that help senders improve deliv­er­abil­ity, and I’ll talk about them in my next post.

Note: this is the first post in a series on email delivery.

Thanks to the efforts of spammers around the globe, it’s increas­ingly diffi­cult to send mail to a Hotmail or Yahoo! address without landing in the spam folder. Over the coming weeks, I’ll attempt to share some of the lessons I’ve learned about how startups can improve their email deliv­ery. I hope that others might contribute their tips as well.

Do you actually need to send your own mail?

Going through the all the steps to get mail deliv­ered from your servers properly can take a long time – a matter of weeks or months, depending on your needs. You probably need your own machine if:

• You send user-generated invites, OR
• You send large volumes of mail (ie, greater than around 1,000 per day)

You’ll save yourself time and money by outsourcing your delivery if you don’t meet the above criteria.

If you can outsource, try these

• AuthSMTP – Offers secure SMTP servers to send your mail with plans priced from $2 / month and up. Great for low-volume senders – reason­ably cheap, and it lets you avoid the headaches that this blog series deals with.
  
  On the downside, you pay for a year in advance, though if you’re sending very high volumes (> 50k a month or so), you can contact them to negotiate a shorter term. Moreover, your mail can’t get marked as spam too often by end-users, as AuthSMTP has very strict usage policies.

• Google Apps – If you’re just getting started, you can send mail through an authen­ti­cated Google Apps mail account. Even the free versions of Google Apps allow you to do this; the caveat is that you can only send to around 500 different recip­i­ents from a single account per day. (As far as I know, this isn’t documented as a hard limit, but it appears to be the general consensus.)

• Newsletter delivery services – there are lots of these, like MailChimp and Constant Contact. They’re not partic­u­larly afford­able for sending user-generated mail, so I haven’t tried any of them, but if anyone has recom­men­da­tions please feel free leave them in the comments.

Sending your own mail

If you want to “go viral” like it’s 2007, or you send lots of mail, you’ll probably be better off setting up your own server. Here’s what you need to get started:

• Mail transfer agent: Installing an MTA is outside the scope of this blog; there are plenty of HOWTOs for various software packages and platforms out there. This is a nice comparison of the major MTAs. I’m sure everyone has their own favorite and their reasons behind it; mine is Postfixdue to its security and ease of configuration.

• Dedicated machine for sending mail: You need at least one dedicated box/IP for deliv­ering mail. Each IP doesn’t have to be its own box/vir­tual server, but you should leave yourself some vertical upgrade room so that if your mail volume increases, you can still maintain the same IP. Your machine needs to be acces­sible by you and only you/your company – email whitelisting programs will want to verify your exclu­sive access with your ISP (more on this later).

You should provi­sion a mail server sooner rather than later if you don’t have one already. Sender reputa­tion is both domain-based and IP based, so getting a positive sending history started on your new IP helps estab­lish you as a “good guy.”

Most ISPs recom­mend that you deliver different classes of mail from separate IPs, so that if one IP gets black­listed it doesn’t affect your others. For example, you might want to send your user invita­tions from a different IP than your registration emails.

Next steps

Hopefully, with a bit of work, you can get your own mailserver up and running on its own IP. Unfor­tu­nately, that’s just the begin­ning. Up next we’ll talk about DNS-based anti-spam techniques, like Sender ID, DomainKeys, and DKIM, followed by bounce processing and ISP-specific programs and whitelists.

Got a question or a sugges­tion? Please leave a comment.

A short state­ment of purpose:

I’m restarting a blog because I’ve been working on a web startup for 18 months or so, creating a total of four or five different Web projects, and I haven’t kept a good record of my work or any lessons learned from those projects. This blog will be an attempt to share a small amount of what I’ve learned, as a modest way of repaying all those folks from whom I’ve learned something.

Here’s to (hope­fully) contributing something of value.

Of course there are all different kinds of freedom, and the kind that is most precious you will not hear much talk about much in the great outside world of wanting and achiev­ing… The really impor­tant kind of freedom involves atten­tion and aware­ness and disci­pline, and being able truly to care about other people and to sacri­fice for them over and over in myriad petty, unsexy ways every day.

- David Foster Wallace, in a 2005 commence­ment speech at Kenyon College.

Please read this speech if you haven’t already. I can’t admit to having read much of Wallace’s work before his death, but I have been profoundly moved by his words. In a talk that must have lasted less than 30 minutes, Wallace hit upon something that I have been trying to artic­u­late, in one way or another, for many years.

There are two funda­men­tally different modes of living. Wallace describes the first, perhaps more common, mode as the “de­fault setting, hard-wired into our boards at birth,” which is “to be deeply and liter­ally self-centered and to see and inter­pret every­thing through this lens of self.” Its opposite is a constant internal struggle towards self-awareness and empathy, part of which involves being sometimes painfully aware of how often one falls short of that goal.

I believe this divide explains many modern phenom­ena: why driving sucks, for example, or much polit­ical rhetoric this season.