Dave Zohrob

Note: this is the third post in a series on email delivery.

After ignoring this series for a couple of months, Garry from Posterous submitted my original post toHacker News a few days ago. Since a few people seem to have found it useful, I’ve got a renewed motiva­tion for hammering out a few more posts.

All of the major ISPs and email providers have some propri­etary methods of inter­acting with large-volume email senders. Most include a feedback loop, which notifies email senders when a recip­ient marks a message as spam. This post will help you get set up with the major web-based email provider­s—Mi­crosoft, Yahoo!, AOL, and GMail.

Microsoft Hotmail & Live Mail

Hotmail is still one of the largest email services out there, even though if you live in the Valley you probably know zero people who use it. Because it is such a large attack surface, Microsoft­’s spam filter (called SmartScreen) is pretty aggres­sive, especially with mail coming from IP addresses with no sending history. Fortu­nately, they’ve imple­mented two major programs to help legit­i­mate email senders get into the inbox.

• Smart Network Data Services — SNDS gives you deliv­er­ability infor­ma­tion based on the IP of your mailserver. You’ll need a Hotmail account to use as your SNDS login. Fill out the access request form, and you’ll receive a verifi­ca­tion link in an email to abuse or postmaster@yourdomain. 
  
  SNDS requires a minimum volume of about 100 emails per day in order to give you any data. Once your IP is activated, you’ll get infor­ma­tion on how your emails are being treated by Hotmail, including data on the number of unknown addresses, complaint rates, spam trap hits, and more. See the SNDS FAQ for more detailed information.

• Junk Mail Reporting Program (JMRP) – This is Hotmail’s feedback loop program. Fill out thisMicrosoft Support questionnaire after signing up for SNDS above. There will be a legal document signing process that you’ll have to go through, but once you’re set up, you’ll receive an email to a speci­fied address for every spam complaint from a Hotmail user. You should remove complaining users from your sending lists in order to preserve your sender reputation.

The Postmaster Services home page has more infor­ma­tion on both programs.

Yahoo!

Yahoo! Mail is the largest email provider in the US. Unfor­tu­nately, their feedback loop program has been closed to new appli­ca­tions for a year or more—see this FAQ answer on the Yahoo! postmaster site for more details.

As of January 2009, your best bet is to first read Yahoo!’s sending best practices, make sure you comply, and then fill out the Bulk Sender Form. If you only send double-opt-in mail (ie, you’re not sending user-generated invita­tions), you may be eligible for their Whitelist as well.

If you happen to have any infor­ma­tion on the Yahoo! feedback loop process, please leave a comment!

AOL

Plenty of low-tech US users still use AOL mail. They make very clear their email sending best practices, much like the other providers. Sign up for the feedback loop here. If you only send double opt-in mail (again, no user-generated invites), you can sign up for their whitelist.

GMail

As befits Google’s philos­o­phy, there’s very little human input involved in their spam filter­ing. Here are the GMail bulk sending guidelines. If you do run into deliv­er­ability issues here, there is a support form to contact them as well.

I person­ally haven’t had many issues getting email deliv­ered to Google — as long as you set up SPF and DKIM correctly (see my previous post in this series) and keep your complaint rates low, your mail should go into the inbox.

Funny how quickly six weeks can pass without an update; I’ve been meaning to finish my series on e-mail deliv­ery, and maybe write more about some of the music and books I’ve been appreciating lately.

This passage from The Love of a Good Woman by Alice Munro caught me today:

She still says this every once in awhile.

“What I remember most is that I could­n’t touch you and wondering if you understood.”

Karin says yes. She under­stood. What she doesn’t bother to say is that back then she thought Rosemary’s sorrow was absurd. It was as if she was complaining about not being able to reach across a conti­nent. For that was what Karin had felt she had become—­some­thing immense and shimmering and suffi­cient, ridged up in pain in some places and flattened out, other­wise, into long dull distances. Away off at the edge of this was Rosemary, and Karin could reduce her, any time she liked, into a config­u­ra­tion of noisy black dots. And she herself—Kar­in—­could be stretched out like this and at the same time shrunk into the middle of her terri­tory, as tidy as a bead or a ladybug.

I find myself returning to Munro’s stories even though I’m in the middle of reading a couple of other books right now, both for their emotional immediacy and how well their length and struc­ture lends itself to my current schedule.

In college all of them had studied the putative effects of deraci­na­tion, which were angst and anomie, those dull horrors of the modern world. They had been examined on the subject, had rehearsed bleak and porten­tous philoso­phies in term papers, and they had done it with the earnest suspen­sion of doubt that afflicts the highly educa­ble. And then their return to the pays natal, where the same old willows swept the same ragged lawns, where the same old prairie arose and bloomed as negli­gence permit­ted. Home. What kinder place could there be on earth, and why did it seem to them all like exile? Oh, to be passing anony­mously through an imper­sonal landscape! Oh, not to know every stump and stone, not to remember how the fields of Queen Anne’s lace figured in the childish happi­ness they had offered to their father’s hopes, God bless him.

Passages like this make me think that Marilynne Robinson may be my new favorite author. Home is a wonderful book, beautiful and moving and written with such evoca­tive, precise language. I can’t recom­mend it enough. But if my recommen­da­tion hasn’t convinced you, please read James Wood’s excellent piece in the New Yorker about Robinson and her work.

Note: this is the second post in a series on email delivery.

So you’ve just set up your own email server and you’re ready to send your first message. You type in your address, click “Send,” and wait. And wait. Chances are, the message will end up in your spam folder, if it shows up at all.

There are a few existing proto­cols that will greatly improve your chances of getting into the inbox. They are, in increasing order of complexity:

• Reverse DNS
• SPF / Sender ID
• DomainKeys / DKIM

Reverse DNS

The easiest way you can improve your deliv­er­ability is to make sure that a reverse DNS lookup on your mail server returns the server’s hostname. Usually you have to ask your ISP to set this up.

Let’s look at a Digg mailserver for an example. From a UNIX prompt, you can type dig mail.digg.comto get this result (or something like it):

mail.digg.com. 1139 IN A 64.191.203.36

Now, if you check the reverse DNS record for that IP address with dig -x 209.191.118.103, you’ll see that the reverse DNS matches up:

36.203.191.64.in-addr.arpa. 3572 IN PTR mail.digg.com.

Reverse DNS is required for good deliv­ery. Without it, ISPs may reject the message. Hotmail might accept the mail but not deliver it (it just vanish­es), or just throw your mail into the spam folder.

SPF / Sender ID

Sender Policy Framework is a simple protocol for speci­fying which servers are allowed to send mail for a partic­ular domain. All you have to do is set up a TXT DNS record according to a simple format.

Let’s look at Reddit as an example. A quick dig reddit.com txt will pull up their TXT:

reddit.com. 212 IN TXT "v=spf1 mx ip4:208.96.53.70 mx:mail.reddit.com ~all"

Tearing the SPF record apart, we get:

• v=spf1 identi­fies an SPF specification
• mx allows the mail servers for reddit.com to send mail (those listed in its MX records)
• ip4:208.96.53.70 autho­rizes a single IP to send mail
• mx:mail.reddit.com allows any MX servers for mail.red­dit.com to send mail as well
• ~all speci­fies that any machi­nes/ad­dresses NOT listed here are not autho­rized to send mail. The~ indicates a “tran­si­tional mode” – once you’re done testing your record, use a dash (-all).

The specification is here, and this tool will help you set up your own record. If you’re sending invites or user-generated mail, check out this OpenSPF best practices page for infor­ma­tion on how to make your invita­tion show up as “From:” a partic­ular user, but ensure that any bounces come back to your mailserver.

A quick note on terminology: Microsoft has a technology that they call Sender ID, which performs the same function as Sender ID but with a slightly different format. For most practical purposes, a valid SPF record is a valid Sender ID record, and when a Microsoft service talks about Sender ID, just read it as SPF. For the curious, the Wikipedia article on Sender ID has more details.

DomainKeys and DKIM

Both DomainKeys and DKIM (DomainKeys Identi­fied Mail) are DNS-based proto­cols for email authen­ti­ca­tion using a public key speci­fied in your DNS record. Before sending a message, your mailserver “sign­s” the email and puts the result in a header on the message. Any recip­ient can verify that the message origi­nated from your servers by checking the public key against the signature header.

DomainKeys is an older standard created by Yahoo!, which is now being replaced by the DKIM standard. Because they are different standards, and different ISPs only support one or the other, you’ll need to set up both DomainKeys and DKIM signing.

Yahoo! and GMail both give a valid signa­ture high impor­tance in deter­mining whether or not a message is spam, and usually tell the user that a message is authen­ti­cated. For example, GMail shows a “signed-by” field:

Let’s take a peek at a record in the wild. Twitter’s DKIM public key can be found by runningdig default._domainkey.twitter.com txt, which returns:

default._domainkey.twitter.com. 600 IN TXT "g=\;" "k=rsa\;" "t=y\;" "p=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAN+FNJESkUBl+vuJDPsL3RSgYI9Qzlq43+l7Q72pRZRDprrhZTXIi7NdSqy+f9hn" "pet1pKMYMYnCxgmaS3qhUXMCAwEAAQ=="

The longest section of this record is the key itself, after the p=. Some registrars don’t let you store TXT records this long, and if you get an error setting up your public key, you might have to switch to a new DNS provider. You can learn about the other fields from the DKIM Spec.

If you’re using Postfix, DKIMProxy does both DomainKeys and DKIM well, and there are decent step-by-step instruc­tions its homepage. You can also check out the SourceForge DomainKeys page to find software for different MTAs as well as testing tools.

Next steps…

Reverse DNS, SPF, and DKIM should definitely help your deliv­er­ability rates – if you weren’t getting through at all, you should at least be hitting the Spam folder now. However, there’s more work ahead. Most ISPs have programs that help senders improve deliv­er­abil­ity, and I’ll talk about them in my next post.

Note: this is the first post in a series on email delivery.

Thanks to the efforts of spammers around the globe, it’s increas­ingly diffi­cult to send mail to a Hotmail or Yahoo! address without landing in the spam folder. Over the coming weeks, I’ll attempt to share some of the lessons I’ve learned about how startups can improve their email deliv­ery. I hope that others might contribute their tips as well.

Do you actually need to send your own mail?

Going through the all the steps to get mail deliv­ered from your servers properly can take a long time – a matter of weeks or months, depending on your needs. You probably need your own machine if:

• You send user-generated invites, OR
• You send large volumes of mail (ie, greater than around 1,000 per day)

You’ll save yourself time and money by outsourcing your delivery if you don’t meet the above criteria.

If you can outsource, try these

• AuthSMTP – Offers secure SMTP servers to send your mail with plans priced from $2 / month and up. Great for low-volume senders – reason­ably cheap, and it lets you avoid the headaches that this blog series deals with.
  
  On the downside, you pay for a year in advance, though if you’re sending very high volumes (> 50k a month or so), you can contact them to negotiate a shorter term. Moreover, your mail can’t get marked as spam too often by end-users, as AuthSMTP has very strict usage policies.

• Google Apps – If you’re just getting started, you can send mail through an authen­ti­cated Google Apps mail account. Even the free versions of Google Apps allow you to do this; the caveat is that you can only send to around 500 different recip­i­ents from a single account per day. (As far as I know, this isn’t documented as a hard limit, but it appears to be the general consensus.)

• Newsletter delivery services – there are lots of these, like MailChimp and Constant Contact. They’re not partic­u­larly afford­able for sending user-generated mail, so I haven’t tried any of them, but if anyone has recom­men­da­tions please feel free leave them in the comments.

Sending your own mail

If you want to “go viral” like it’s 2007, or you send lots of mail, you’ll probably be better off setting up your own server. Here’s what you need to get started:

• Mail transfer agent: Installing an MTA is outside the scope of this blog; there are plenty of HOWTOs for various software packages and platforms out there. This is a nice comparison of the major MTAs. I’m sure everyone has their own favorite and their reasons behind it; mine is Postfixdue to its security and ease of configuration.

• Dedicated machine for sending mail: You need at least one dedicated box/IP for deliv­ering mail. Each IP doesn’t have to be its own box/vir­tual server, but you should leave yourself some vertical upgrade room so that if your mail volume increases, you can still maintain the same IP. Your machine needs to be acces­sible by you and only you/your company – email whitelisting programs will want to verify your exclu­sive access with your ISP (more on this later).

You should provi­sion a mail server sooner rather than later if you don’t have one already. Sender reputa­tion is both domain-based and IP based, so getting a positive sending history started on your new IP helps estab­lish you as a “good guy.”

Most ISPs recom­mend that you deliver different classes of mail from separate IPs, so that if one IP gets black­listed it doesn’t affect your others. For example, you might want to send your user invita­tions from a different IP than your registration emails.

Next steps

Hopefully, with a bit of work, you can get your own mailserver up and running on its own IP. Unfor­tu­nately, that’s just the begin­ning. Up next we’ll talk about DNS-based anti-spam techniques, like Sender ID, DomainKeys, and DKIM, followed by bounce processing and ISP-specific programs and whitelists.

Got a question or a sugges­tion? Please leave a comment.

A short state­ment of purpose:

I’m restarting a blog because I’ve been working on a web startup for 18 months or so, creating a total of four or five different Web projects, and I haven’t kept a good record of my work or any lessons learned from those projects. This blog will be an attempt to share a small amount of what I’ve learned, as a modest way of repaying all those folks from whom I’ve learned something.

Here’s to (hope­fully) contributing something of value.

Of course there are all different kinds of freedom, and the kind that is most precious you will not hear much talk about much in the great outside world of wanting and achiev­ing… The really impor­tant kind of freedom involves atten­tion and aware­ness and disci­pline, and being able truly to care about other people and to sacri­fice for them over and over in myriad petty, unsexy ways every day.

- David Foster Wallace, in a 2005 commence­ment speech at Kenyon College.

Please read this speech if you haven’t already. I can’t admit to having read much of Wallace’s work before his death, but I have been profoundly moved by his words. In a talk that must have lasted less than 30 minutes, Wallace hit upon something that I have been trying to artic­u­late, in one way or another, for many years.

There are two funda­men­tally different modes of living. Wallace describes the first, perhaps more common, mode as the “de­fault setting, hard-wired into our boards at birth,” which is “to be deeply and liter­ally self-centered and to see and inter­pret every­thing through this lens of self.” Its opposite is a constant internal struggle towards self-awareness and empathy, part of which involves being sometimes painfully aware of how often one falls short of that goal.

I believe this divide explains many modern phenom­ena: why driving sucks, for example, or much polit­ical rhetoric this season.