7 January 2015
DKIM (DomainKeys Identified Mail) is another type of email deliverability record that helps recipient servers be confident that you authorized any given email. DKIM uses public-key-cryptography to mathematically sign important parts of your messages. This post is a deep dive into how it works and what it's good for.
Why do we need DKIM?
Like SPF, DKIM protects your domain against spammers and phishers by validating your legitimate mail. Mail that purports to come from you and doesn't have a DKIM signature is more suspicious and more likely to be put into recipient's spam folder.
DKIM uses the DNS in a similar way as SPF. When you deploy DKIM, you insert keys into your DNS records at specifc URLs, thus proving that you control your DNS records.
An Example
DKIM has more moving parts than SPF. Specifically, there are:
- Records in your DNS settings that contain public keys
- Private keys on your mail server
- Cryptographic signatures embedded in your messages
We're going to explore each one of these in turn.
DNS Records
DKIM public keys are stored as TXT records on your domain, under the subdomain _domainkey. As an example, here's the DKIM key that Mandrill uses when sending as petekeen.net:
$ dig +short mandrill._domainkey.petekeen.net txt
"v=DKIM1\; k=rsa\; p=RSA_PUBLIC_KEY>\;"
DKIM records are always key-value separated by semicolons (the backslashes come from the dig tool, they don't actually need to be there). In this case there are three parts. v=DKIM1 says that this is a DKIM version 1 key. k=rsa says that this is an RSA public key. p=RSA_PUBLIC_KEY_DATA is the actual public key (I removed the data just because it's so big. Run the dig command to see the data).
Unless you're running your own mail server, you'll almost always get the value for this record from your email provider. It'll be buried in your account settings, usually under a header like "Verified Domains".
Private Keys
The private key that corresponds to the public key in your DNS lives on the mail server. If you're using an email service provider like Mandrill, Postmark, or Mailgun they handle this for you. If you're running your own mail server you'll need to handle it yourself, which is out of scope for this post.
Embedded Message Signature
When a mail server wants to send a DKIM-signed message, it first calculates a cryptographic signature for the body and certain headers. Here's an example from a message I sent a few days ago:
DKIM-Signature:
v=1;
a=rsa-sha1;
c=relaxed/relaxed;
s=mandrill;
d=petekeen.net;
h=From:Subject:To:Message-Id:Date:MIME-Version:Content-Type;
i=pete@petekeen.net;
bh=82iZmY7kCbFDunaEckImLSxqHv8=;
b=IQK/KMfy9xVjTU2TEIkWVaajqjmwdc9xnc3yByC6dZQjeFmYD3Rvaeu6lct44vBLymxkdT5Po7G6
b5Li5KWjcBZJ95L6ur1DaBZDTN2E6aVwd+5cQ4zFm4MXhMC6uAssS3+eUK+ZFteDLgkmns+q/Gbt
5bqJZuixpEhqgM4exLI=
This is again a set of key-value pairs, just like the DNS record. It says that the headers in the h= (plus the DKIM-Signature header itself), combined with the hash of the email body in bh=, when signed by the private key that matches the public key at mandrill._domainkey.petekeen.net, produce the cryptographic signature in the b= field.
(You may be wondering how DKIM can sign a header that includes it's own signature. The answer is that the b= field is treated as if it were an empty string when calculating the signature.)
The sending mail server embeds this DKIM-Signature header into the message. When a DKIM-compatible mail server receives a message with a signature, it downloads the public key specified in the header and processes the given signature against the message. If they match, the message is authentic. If they don't match, something's wrong. Either way, a new header is attached to the message named Authentication-Results that tells servers and spam filters further along how to handle the message.
Limits
Unlike SPF, DKIM doesn't have a built-in specification for how to handle failing signatures. It's up to each receiving server how to handle it. Some send a bounce message, some just attach the Authentication-Results header and let other servers handle it.
DKIM also explicitly doesn't handle what to do when a message has no signature at all. For that, we need another email deliverability record named DMARC. See my article Fix Your Email Deliverability with DMARC for more details on how to handle it.
Tagged:
Email
DNS
_evergreen
Read More
5 January 2015
Sender Policy Framework (SPF) is a type of email deliverability record that helps servers that receive email verify that the sender is allowed to send. It's been around for a few years and has been taken up by every large email provider. This post is a deep dive into how it works and what it's good for.
Why do we need SPF?
The protocol that we use to send email from server to server, SMTP, allows anyone to set anything as their From or MAIL FROM (the envelope-from) address. Effectively, if I know your email address I can send messages as if I were you, and nobody would be the wiser unless they looked very closely at the message headers and compared them with previous messages from you.
SPF is a standard for declaring what servers can actually send on your behalf. An SPF record is attached to your domain name as a DNS TXT record. Domains that have SPF are less likely to be used by spammers and phishers, which means genuine messages coming from those domains will be given higher reputation with receiving servers.
An Illustrative Example
Here is bugsplat.info's SPF record:
$ dig +short bugsplat.info txt
v=spf1
ip4:104.131.72.15
ip4:192.241.250.244
include:servers.mcsv.net
include:_spf.google.com
include:spf.messagingengine.com
~all
An SPF record consists of a version tag (v=spf1) followed by one or more mechanisms. There's a bunch of different mechanisms, along with a whole macro system that you can use to make things really complicated for yourself, but we're going to stick to the basics for now.
Mechanisms are matched from left to right, until something returns true. At a high level, this says that if the email is coming from one of those two IP's, it's good. In addition, any of the IPs listed in those includeed polices are also good. The google.com one is pretty obvious, that means that Google's GMail servers can send as bugsplat.info. servers.mcsv.net is Mailchimp's address, and spf.messagingengine.com is for Fastmail.
The last mechanism, ~all, will match everything that hits it. The tilde is a "qualifier" that means to fall through with a "soft fail" status. A soft fail basically says that the message came from an unknown source, but don't take action on it. The default qualifier is +, which means "accept the message". For the ip4 and include mechanisms above, if they match the sending server then the message will be accepted.
include is actually sort of a misnomer, because the contents of the referenced record doesn't actually get included anywhere. Including another record means: evaluate the email against this entire other SPF declaration. If it matches, then return the qualifier on the include mechanism (by default, +). If it doesn't, continue on with the rest of the mechanisms. The only thing that causes a match when evaluating an include is a +. Soft fail, normal fail, and neutral all cause the include to not match.
Limits
SPF records are limited to 10 total DNS lookups. Every include generates a DNS lookup, as do many other mechanisms including ptr and a. The only common mechanisms that don't are all, ip4, and ip6. This limit, along with a practical limit to DNS TXT records of 450 characters, means that adding a new mail provider to your setup is actually a big deal. You have to be careful to not exceed either of those limits, lest receiving servers start throwing permanent errors.
SPF also doesn't give much direction on what to actually do with a message that fails the checks. Generally your records should be set up to soft-fail, which means receiving servers shouldn't actually take any action other than maybe giving the message a higher spam rating. There's a newer, related standard named DMARC which you can use to actually declare a comprehensive policy. See my article Fix Your Deliverability with DMARC for more details on how to work it.
If you're interested in more SPF mechanisms the official SPF webpage has all the details. This HOWTO goes into even more depth, along with an explanation of the SPF macro system, if you're so inclined.
P.S. Could your email deliverability use a boost? Mail Rep will help your business improve your deliverability in just two weeks. Click here to read about Mail Rep.
Tagged:
Email
DNS
_evergreen
Read More
31 December 2014
Everybody knows what email is. You click "Compose", fill in the recipient's email address, write your message, maybe give it a pithy subject, and hit "Send". Some time later your recipient opens their email and reads your message. Simple, right?
There are an awful lot of interesting things happening between "Send" and "Some time later". This post is an overview of what happens when you hit "send", and how your message makes its way to your recipient.
Fundamentals
Under the hood, an email is just a text file. When you send an email, your email client (like Gmail, Mail.app, or Outlook) takes your text, injects some information into "headers" (formatted text before the start of your text), and hands it off to a mail server. Here's an example of a simple composed email:
From: Pete Keen <pete@petekeen.net>
To: Joe Example <joe@example.com>
Date: Sun, 28 Dec 2014 09:52:03 -0600
Subject: This is a simple message
Here's some text in the message. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis interdum in neque sed tincidunt. Nullam sed auctor libero, sed facilisis ligula. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Aliquam sit amet dui a ligula ultrices porta quis vel mi. Nulla ac urna augue. Donec euismod tristique odio eget convallis. Cras ac quam vel sapien pharetra luctus. Aliquam sem eros, auctor non fringilla sed, viverra non turpis. Sed suscipit egestas posuere. In hac habitasse platea dictumst. Nulla facilisi.
As you can see, there's really not much to a simple email. Messages can get far more complex, with multi-part HTML messages and attachments, but fundamentally they're just formatted text files with headers.
Mail Transmission
After your client (mail user agent) composes your message it hands it off to a mail server (mail transfer agent), which will hand it to at least one other server at some point before it ultimately ends up in your recipient's inbox. Mail transmission happens via the Simple Message Transport Protocol (SMTP), a simple text-based protocol. Here's a simple SMTP transaction (blue lines with right arrows are what we send to the server, black lines with left arrows are what we receive from the server):
← 220 web01.bugsplat.info ESMTP Postfix
→ HELO client.bugsplat.info
← 250 web01.bugsplat.info
→ MAIL FROM: test@bugsplat.info
← 250 2.1.0 Ok
→ RCPT TO: pete@bugsplat.info
← 250 2.1.5 Ok
→ DATA
← 354 End data with .
→ From: test@bugsplat.info
→ To: pete@bugsplat.info
→ Subject: this is a test
→
→ Example test.
→ .
← 250 2.0.0 Ok: queued as CD76D606BB
→ QUIT
← 221 2.0.0 Bye
The transaction starts with an announcment, which we respond to with a HELO command specifying who we are. The server replies with its domain name.
Next, we tell the server who the message is from, who it should ultimately be delivered to, then we actually send the message. An important concept here is that the MAIL FROM and RCPT TO commands are separate from the actual contents of the message and the From and To headers. They can be completely different. This is actually how the "BCC" function in email is implemented, the addresses that are BCC'd are actually sent the message with their address in RCPT TO, but the primary address in the To header.
Each time a mail server receives a message it adds a Received header with its hostname, the hostname of the machine who sent it, the exact time, and some other information, generally to the top of the stack of headers. Here's the Received headers from the message we just sent above:
Received: by 10.140.147.132 with SMTP id 126csp2632885qht;
Sun, 28 Dec 2014 09:44:19 -0800 (PST)
Received: from web01.bugsplat.info (web01.bugsplat.info. [192.241.250.244])
by mx.google.com with ESMTPS id kg1si49786991pad.162.2014.12.28.09.44.17
for <peter.keen@gmail.com>
(version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
Sun, 28 Dec 2014 09:44:18 -0800 (PST)
Received: from client.bugsplat.info (sales02.bugsplat.info [104.131.72.15])
by web01.bugsplat.info (Postfix) with SMTP id CD76D606BB
for <pete@bugsplat.info>; Sun, 28 Dec 2014 17:43:19 +0000 (UTC)
In diagram form, this is what the flow looks like:
Reading the headers from the bottom (and the diagram from the top), sales02.bugsplat.info connected to web01.bugsplat.info (lying about who it actually was) and sent a message destined for pete@bugsplat.info. web01 is configured to forward that address to my GMail address, so it connects to Google's mail server and passes along the message using the exact same protocol as shown above. Within GMail there's one more hop to the server that handles my account. Each time an email message is sent from server to server, SMTP is involved.
Lots of other information lives in the headers, including a unique message ID, timestamps, email addresses where you can send complaints, and cryptographic signatures.
You're probably wondering how web01.bugsplat.info found the GMail server in the first place. How did it know where to send the message? It looked at the DNS, where Google has set up MX (mail exchange) records for the gmail.com domain. Here's what those look like:
$ dig +short gmail.com mx
5 gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
20 alt2.gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.
40 alt4.gmail-smtp-in.l.google.com.
Mail servers are tried in priority order, where the lower number has higher priority. If a receiving mail server doesn't respond the sender will try the next in the list, and if it runs out of servers to try it will hold onto the message for awhile before trying again. You can look up your company's MX records just as easily with the dig tool:
$ dig +short yourdomain.com mx
For more information on DNS, check out DNS: The Good Parts.
Spam
Spam is sort of a nebulous term, but it boils down to unwanted messages. Things you didn't sign up for, don't want, and/or are actively harmful.
Spam has been around for a long time. In fact, the first recognized unwanted message on the internet was sent in 1978, before SMTP was even around (it was an advertisement for a presentation by Digital Equipment Corproation). Since then it's morphed into an actual business model. Drugs, merchandise, insurance offers, and everything else under the sun is advertised via spam messages, sent out by the billions by rogue mail servers. In addition to real (but unwanted) products, the same techniques get used to send out malicious messages like phishing attacks.
The protocol underlying email, named SMTP (Simple Message Transport Protocol) is from a much more civilized era. There's very little sanity checking or verification in the base protocol, which means it's trivial to forge various parts of a message, including headers. If I know your address I can trivially craft a message that, by all outward appearances, looks like it came from you.
Transactional vs Newsletters
There are two broad swaths of email that businesses send today, other than personal correspondence. Transactional is generally generated in response to events within your application and goes to one person, one at a time.
On the other hand, newsletters are considered "bulk" email. They go from you to a list of people, all at the same time. This doesn't mean they're spam, just that you send a large number of almost-identitcal messages at the same time.
Email service providers distinguish their handling of these different types of messages. For example, spam rules are applied more stringently to messages that they detect as bulk. Depending on the keywords and general format of your messages, GMail might automatically decide to put your bulk email into their "Promotions" tab.
Trust
Over the years, technologies have come around that help to prevent forgery and help the big providers like Gmail and Yahoo trust your messages, both transactional and newsletters. Those are:
- SPF, Sender Policy Framework. Lets the world know what servers are authorized to send mail as your domain.
- DKIM, Domain Key Idenitifed Mail. Allows sending email servers to cryptographically sign messages, verifiying that you authorized them to send email as you.
- DMARC, Domain-based Message Authentication, Reporting, and Conformance. Specifies a policy for recipient servers, telling them what to do if a message fails SPF or DKIM checks.
I went into these somewhat in Fix Your Email Deliverability with DMARC. In future posts I'll be taking a deep dive into each one, talking about its history, what its good for, and how to best deploy it.
Tagged:
Email
DNS
_evergreen
Read More