Articles tagged with 'dns'

DKIM Deep Dive

7 January 2015

DKIM (DomainKeys Identified Mail) is another type of email deliverability record that helps recipient servers be confident that you authorized any given email. DKIM uses public-key-cryptography to mathematically sign important parts of your messages. This post is a deep dive into how it works and what it's good for.

Why do we need DKIM?

Like SPF, DKIM protects your domain against spammers and phishers by validating your legitimate mail. Mail that purports to come from you and doesn't have a DKIM signature is more suspicious and more likely to be put into recipient's spam folder.

DKIM uses the DNS in a similar way as SPF. When you deploy DKIM, you insert keys into your DNS records at specifc URLs, thus proving that you control your DNS records.

An Example

DKIM has more moving parts than SPF. Specifically, there are:

  • Records in your DNS settings that contain public keys
  • Private keys on your mail server
  • Cryptographic signatures embedded in your messages

We're going to explore each one of these in turn.

DNS Records

DKIM public keys are stored as TXT records on your domain, under the subdomain _domainkey. As an example, here's the DKIM key that Mandrill uses when sending as

$ dig +short txt
"v=DKIM1\; k=rsa\; p=RSA_PUBLIC_KEY>\;"

DKIM records are always key-value separated by semicolons (the backslashes come from the dig tool, they don't actually need to be there). In this case there are three parts. v=DKIM1 says that this is a DKIM version 1 key. k=rsa says that this is an RSA public key. p=RSA_PUBLIC_KEY_DATA is the actual public key (I removed the data just because it's so big. Run the dig command to see the data).

Unless you're running your own mail server, you'll almost always get the value for this record from your email provider. It'll be buried in your account settings, usually under a header like "Verified Domains".

Private Keys

The private key that corresponds to the public key in your DNS lives on the mail server. If you're using an email service provider like Mandrill, Postmark, or Mailgun they handle this for you. If you're running your own mail server you'll need to handle it yourself, which is out of scope for this post.

Embedded Message Signature

When a mail server wants to send a DKIM-signed message, it first calculates a cryptographic signature for the body and certain headers. Here's an example from a message I sent a few days ago:


This is again a set of key-value pairs, just like the DNS record. It says that the headers in the h= (plus the DKIM-Signature header itself), combined with the hash of the email body in bh=, when signed by the private key that matches the public key at, produce the cryptographic signature in the b= field.

(You may be wondering how DKIM can sign a header that includes it's own signature. The answer is that the b= field is treated as if it were an empty string when calculating the signature.)

The sending mail server embeds this DKIM-Signature header into the message. When a DKIM-compatible mail server receives a message with a signature, it downloads the public key specified in the header and processes the given signature against the message. If they match, the message is authentic. If they don't match, something's wrong. Either way, a new header is attached to the message named Authentication-Results that tells servers and spam filters further along how to handle the message.


Unlike SPF, DKIM doesn't have a built-in specification for how to handle failing signatures. It's up to each receiving server how to handle it. Some send a bounce message, some just attach the Authentication-Results header and let other servers handle it.

DKIM also explicitly doesn't handle what to do when a message has no signature at all. For that, we need another email deliverability record named DMARC. See my article Fix Your Email Deliverability with DMARC for more details on how to handle it.

Tagged: Email  DNS  _evergreen 

Read More

SPF: Sunscreen for your Email

5 January 2015

Sender Policy Framework (SPF) is a type of email deliverability record that helps servers that receive email verify that the sender is allowed to send. It's been around for a few years and has been taken up by every large email provider. This post is a deep dive into how it works and what it's good for.

Why do we need SPF?

The protocol that we use to send email from server to server, SMTP, allows anyone to set anything as their From or MAIL FROM (the envelope-from) address. Effectively, if I know your email address I can send messages as if I were you, and nobody would be the wiser unless they looked very closely at the message headers and compared them with previous messages from you.

SPF is a standard for declaring what servers can actually send on your behalf. An SPF record is attached to your domain name as a DNS TXT record. Domains that have SPF are less likely to be used by spammers and phishers, which means genuine messages coming from those domains will be given higher reputation with receiving servers.

An Illustrative Example

Here is's SPF record:

$ dig +short txt

An SPF record consists of a version tag (v=spf1) followed by one or more mechanisms. There's a bunch of different mechanisms, along with a whole macro system that you can use to make things really complicated for yourself, but we're going to stick to the basics for now.

Mechanisms are matched from left to right, until something returns true. At a high level, this says that if the email is coming from one of those two IP's, it's good. In addition, any of the IPs listed in those includeed polices are also good. The one is pretty obvious, that means that Google's GMail servers can send as is Mailchimp's address, and is for Fastmail.

The last mechanism, ~all, will match everything that hits it. The tilde is a "qualifier" that means to fall through with a "soft fail" status. A soft fail basically says that the message came from an unknown source, but don't take action on it. The default qualifier is +, which means "accept the message". For the ip4 and include mechanisms above, if they match the sending server then the message will be accepted.

include is actually sort of a misnomer, because the contents of the referenced record doesn't actually get included anywhere. Including another record means: evaluate the email against this entire other SPF declaration. If it matches, then return the qualifier on the include mechanism (by default, +). If it doesn't, continue on with the rest of the mechanisms. The only thing that causes a match when evaluating an include is a +. Soft fail, normal fail, and neutral all cause the include to not match.


SPF records are limited to 10 total DNS lookups. Every include generates a DNS lookup, as do many other mechanisms including ptr and a. The only common mechanisms that don't are all, ip4, and ip6. This limit, along with a practical limit to DNS TXT records of 450 characters, means that adding a new mail provider to your setup is actually a big deal. You have to be careful to not exceed either of those limits, lest receiving servers start throwing permanent errors.

SPF also doesn't give much direction on what to actually do with a message that fails the checks. Generally your records should be set up to soft-fail, which means receiving servers shouldn't actually take any action other than maybe giving the message a higher spam rating. There's a newer, related standard named DMARC which you can use to actually declare a comprehensive policy. See my article Fix Your Deliverability with DMARC for more details on how to work it.

If you're interested in more SPF mechanisms the official SPF webpage has all the details. This HOWTO goes into even more depth, along with an explanation of the SPF macro system, if you're so inclined.

P.S. Could your email deliverability use a boost? Mail Rep will help your business improve your deliverability in just two weeks. Click here to read about Mail Rep.

Tagged: Email  DNS  _evergreen 

Read More

Email: The Good Parts

31 December 2014

Everybody knows what email is. You click "Compose", fill in the recipient's email address, write your message, maybe give it a pithy subject, and hit "Send". Some time later your recipient opens their email and reads your message. Simple, right?

There are an awful lot of interesting things happening between "Send" and "Some time later". This post is an overview of what happens when you hit "send", and how your message makes its way to your recipient.


Under the hood, an email is just a text file. When you send an email, your email client (like Gmail,, or Outlook) takes your text, injects some information into "headers" (formatted text before the start of your text), and hands it off to a mail server. Here's an example of a simple composed email:

From: Pete Keen <>
To: Joe Example <>
Date: Sun, 28 Dec 2014 09:52:03 -0600
Subject: This is a simple message

Here's some text in the message. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis interdum in neque sed tincidunt. Nullam sed auctor libero, sed facilisis ligula. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Aliquam sit amet dui a ligula ultrices porta quis vel mi. Nulla ac urna augue. Donec euismod tristique odio eget convallis. Cras ac quam vel sapien pharetra luctus. Aliquam sem eros, auctor non fringilla sed, viverra non turpis. Sed suscipit egestas posuere. In hac habitasse platea dictumst. Nulla facilisi.

As you can see, there's really not much to a simple email. Messages can get far more complex, with multi-part HTML messages and attachments, but fundamentally they're just formatted text files with headers.

Mail Transmission

After your client (mail user agent) composes your message it hands it off to a mail server (mail transfer agent), which will hand it to at least one other server at some point before it ultimately ends up in your recipient's inbox. Mail transmission happens via the Simple Message Transport Protocol (SMTP), a simple text-based protocol. Here's a simple SMTP transaction (blue lines with right arrows are what we send to the server, black lines with left arrows are what we receive from the server):

← 220 ESMTP Postfix
← 250
← 250 2.1.0 Ok
← 250 2.1.5 Ok
← 354 End data with .
→ From:
→ To:
→ Subject: this is a test

→ Example test.
→ .
← 250 2.0.0 Ok: queued as CD76D606BB
← 221 2.0.0 Bye

The transaction starts with an announcment, which we respond to with a HELO command specifying who we are. The server replies with its domain name.

Next, we tell the server who the message is from, who it should ultimately be delivered to, then we actually send the message. An important concept here is that the MAIL FROM and RCPT TO commands are separate from the actual contents of the message and the From and To headers. They can be completely different. This is actually how the "BCC" function in email is implemented, the addresses that are BCC'd are actually sent the message with their address in RCPT TO, but the primary address in the To header.

Each time a mail server receives a message it adds a Received header with its hostname, the hostname of the machine who sent it, the exact time, and some other information, generally to the top of the stack of headers. Here's the Received headers from the message we just sent above:

Received: by with SMTP id 126csp2632885qht;
        Sun, 28 Dec 2014 09:44:19 -0800 (PST)
Received: from ( [])
        by with ESMTPS id kg1si49786991pad.162.2014.
        for <>
        (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Sun, 28 Dec 2014 09:44:18 -0800 (PST)
Received: from ( [])
    by (Postfix) with SMTP id CD76D606BB
    for <>; Sun, 28 Dec 2014 17:43:19 +0000 (UTC)

In diagram form, this is what the flow looks like:

Email Diagram

Reading the headers from the bottom (and the diagram from the top), connected to (lying about who it actually was) and sent a message destined for web01 is configured to forward that address to my GMail address, so it connects to Google's mail server and passes along the message using the exact same protocol as shown above. Within GMail there's one more hop to the server that handles my account. Each time an email message is sent from server to server, SMTP is involved.

Lots of other information lives in the headers, including a unique message ID, timestamps, email addresses where you can send complaints, and cryptographic signatures.

You're probably wondering how found the GMail server in the first place. How did it know where to send the message? It looked at the DNS, where Google has set up MX (mail exchange) records for the domain. Here's what those look like:

$ dig +short mx

Mail servers are tried in priority order, where the lower number has higher priority. If a receiving mail server doesn't respond the sender will try the next in the list, and if it runs out of servers to try it will hold onto the message for awhile before trying again. You can look up your company's MX records just as easily with the dig tool:

$ dig +short mx

For more information on DNS, check out DNS: The Good Parts.


Spam is sort of a nebulous term, but it boils down to unwanted messages. Things you didn't sign up for, don't want, and/or are actively harmful.

Spam has been around for a long time. In fact, the first recognized unwanted message on the internet was sent in 1978, before SMTP was even around (it was an advertisement for a presentation by Digital Equipment Corproation). Since then it's morphed into an actual business model. Drugs, merchandise, insurance offers, and everything else under the sun is advertised via spam messages, sent out by the billions by rogue mail servers. In addition to real (but unwanted) products, the same techniques get used to send out malicious messages like phishing attacks.

The protocol underlying email, named SMTP (Simple Message Transport Protocol) is from a much more civilized era. There's very little sanity checking or verification in the base protocol, which means it's trivial to forge various parts of a message, including headers. If I know your address I can trivially craft a message that, by all outward appearances, looks like it came from you.

Transactional vs Newsletters

There are two broad swaths of email that businesses send today, other than personal correspondence. Transactional is generally generated in response to events within your application and goes to one person, one at a time.

On the other hand, newsletters are considered "bulk" email. They go from you to a list of people, all at the same time. This doesn't mean they're spam, just that you send a large number of almost-identitcal messages at the same time.

Email service providers distinguish their handling of these different types of messages. For example, spam rules are applied more stringently to messages that they detect as bulk. Depending on the keywords and general format of your messages, GMail might automatically decide to put your bulk email into their "Promotions" tab.


Over the years, technologies have come around that help to prevent forgery and help the big providers like Gmail and Yahoo trust your messages, both transactional and newsletters. Those are:

  • SPF, Sender Policy Framework. Lets the world know what servers are authorized to send mail as your domain.
  • DKIM, Domain Key Idenitifed Mail. Allows sending email servers to cryptographically sign messages, verifiying that you authorized them to send email as you.
  • DMARC, Domain-based Message Authentication, Reporting, and Conformance. Specifies a policy for recipient servers, telling them what to do if a message fails SPF or DKIM checks.

I went into these somewhat in Fix Your Email Deliverability with DMARC. In future posts I'll be taking a deep dive into each one, talking about its history, what its good for, and how to best deploy it.

Tagged: Email  DNS  _evergreen 

Read More

Your DNS Provider Should Not Be Your Registrar

1 December 2014

Note: This article references events that happened in December 2014.

Hopefully, by time you're reading this DNSimple will have recovered from their DDoS-powered outage. Today has probably been a terrible day for everybody over there and I'm sure they're ready for a break. While you can't do much to directly defend against DDoS attacks, you can insure yourself against DNS outages.

If you're a DNSimple customer right now or a NameCheap customer several times earlier this year, you know what happens when your DNS service goes out. Your website is inaccessible, emails are probably bouncing, and so are customers and their wallets. It's all around bad news.

The cheapest insurance you can buy is to host your nameservers and your registrar at different companies. That way, if your registrar gets attacked it's no big deal because they're not involved with your day-to-day name resolution, and if your nameservers are attacked you can easily change them. You can't do that if the web interface you need to use is down at the same time as your nameservers.

Splitting your DNS services between two or more companies adds a tiny bit of one-time overhead to setting up a new domain name, but the peace of mind this strategy buys is worth it. Your can be back up and servicing customers at a new DNS provider in as little as five minutes, depending on your registrar, while your previous/primary DNS provider is struggling with an attack for hours.

Personally, I use Amazon's Route53 service as my nameservers and either Gandi (for .io) or Namecheap (for everything else) as my registrars, but you can use whoever you want. You could even use DNSimple as your registrar and Route53 as your nameserver if you want. The point is that you should have at least two wholely separate companies involved.

Tagged: DNS 

Read More

DNS: The Good Parts

19 July 2013

Frequently I come across confusion with domain names. Why doesn't my website work? Why is this stupid thing broken, everything I try fails, I just want it to work!! Invariably the question asker either doesn't know what DNS is or doesn't understand how something fundamental works. More generally, people think that DNS is scary or complicated. This article is an attempt at quelling that fear. DNS is easy once you understand a few basic concepts.

Tagged: DNS  _evergreen 

Read More

How and why I'm not running my own DNS

7 June 2013

A few months ago I posted about how I run my own DNS servers using my virtual private servers and tinydns. Well, it turns out that's not a great idea, for a few reasons. First, because if I mess up I'm entirely shut out of my servers. I tried to turn off a service on them the other day and accidentally turned of the tinydns service instead and it took me ages to get back in. Running DNS on the same machines that handle email and web hosting for almost every piece of my online presence is just way too fragile.

Tagged: DNS  Devops 

Read More

How I run my own DNS servers

31 December 2012

For the longest time I used zoneedit as my DNS provider of choice. All of my important domains were hosted there, they never really did me wrong. A few months back I decided that I wanted to learn how DNS actually works in the real world, though. Like, what does it actually take to run my own DNS servers?

Tagged: DNS  Devops  Email 

Read More