SPF, DMARC, and DKIM: How to Keep Your Email Out of the Spam Folder | Hacker News Reader

SPF, DMARC, and DKIM: How to Keep Your Email Out of the Spam Folder

wpsitecare.com

108 pointswpBenny59 comments

A few tips from setting up SPF/DMARC/DKIM for a SAAS service:

* SPF: limit your record and all includes to 10 DNS lookups (e.g., "A MX include:_spf.google.com" is 3 DNS lookups plus all of the lookups inside the include.

* DMARC: to see a strict reject policy, check out Yahoo:

  $ dig +short -t txt _dmarc.yahoo.com
  "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc_y_rua@yahoo.com\;"

* Mail forwarding: if your app sends mail as the logged-in user, make sure the user's actual email address is not in the FROM address as Yahoo does not authorize you to send FROM: xxxx@yahoo.com

* DMARC emails: use dmarcian.com to parse and process the auto-generated emails

* SPF: use the ~all for your first day of testing and then lock it down to -all after testing is complete

* DKIM: OpenDkim appears to be the most widely supported Linux software package.

* DKIM keys: setup a TXT entry you control and ask client to CNAME it. Then setup key rotation.

On the receiving side: Ensure that your world-facing port 25 smtpd is properly pipelining all incoming emails through openDKIM so that their headers are tagged with openDKIM validation or lack thereof. This can work in conjunction with spamassassin.

https://dmarcian.com/ is a really nice service. Their support was also very helpful while trying to figure out the same things you did.

Since there is a 10 query limit on SPF, if you delegate your SPF to third-parties like Google then your SPF might blow up unexpectedly if they increase the number of records on their side. Dmarcian monitors that for example.

For DKIM, rspamd/rmilter are a great alternative to OpenDKIM if you want to build the DKIM check into your regular spam checks.

One minor downside of rmilter is that it will only sign the headers of mail sent by by authenticated users. This isn't a huge deal, but can be a bit of an irritation.

Rmilter now can sign mail that come from certain networks as well.

That's good to know. I'll have to check that out. The packaged version the FreeBSD port didn't have that last time I looked, but that could just have been me missing something.

Edit: just noticed the 'sign_networks' and 'our_networks' settings. Thanks for that, and thanks for rspamd and rmilter! They're great software!

PostMark also has a helpful (and free) service that parses and summarizes the DMARC reports: https://dmarc.postmarkapp.com/. (And you don't have to be using PostMark to use this service.)

Implemented all of these on a bunch of my domains a few weeks ago. Two cautions related to whether your email can be forwarded by your recipients:

1. SPF -all can break forwarding with DMARC p=reject. If your recipients tend to forward your email, you probably want to stick with SPF ~all (or DMARC p=none) until standards settle (and get widely implemented) around rewriting headers during forwarding. [1]

2. Hotmail and Outlook.com recently introduced a change that breaks forwarding from any DMARC p=reject or p=quarantine sender, through a hotmail.com or outlook.com address, to any recipient MTA that enforces DMARC. There's not really anything you can do about this (as a sender) other than not use DMARC, so hopefully that gets fixed soon. [2]

[1]: https://blogs.msdn.microsoft.com/tzink/2015/07/12/what-is-th...

[2]: https://blogs.msdn.microsoft.com/tzink/2016/05/19/why-does-m...

I run a small Saas company that does mail forwarding as you mentioned above ("send mail as the logged-in user"). If not putting the user's email as FROM, how do you recommend handling it?

I should note, I current set the FROM as "John Smith <donotreply@mydomain.com>" and set Reply-To to their actual email. That said, I've seen a number of clients ignore the Reply To and send email to the donotreply.

You're doing it right. The important part to avoid DMARC filtering is that the actual email address in the From field match your actual sending domain. The display name doesn't matter (for DMARC filtering).

However, there's another problem: Many email clients hide the email address part by default, and only show the display name. So the recipients may not see "donotreply" without some extra clicking. Worse, if that address gets auto-added to their address book, they may accidentally send to it later: the client autocompletes "John Smith", and the user doesn't realize it's actually your donotreply email.

One way to cut down on this is to include your company in the display name. I usually use:

    From: "John Smith via ExampleCo" <donotreply@example.com>

but have also had good results (delivery and open rates) with:

    From: "ExampleCo for John Smith" <donotreply@example.com>

Is there any way to deal with clients ignoring the Reply-To header or is that just an issue we have to deal with?

Most clients seem to respect the Reply-To header, but they're not required to [1], and some popular MS ones in particular seem to ignore it, at least in some cases [2, 3].

There are a bunch of arguments for avoiding "noreply" addresses in the first place [4, 5]. We started using our customer service email as the from/reply-to for password resets and other service emails, with good results.

But that's not a good option for messages you're sending on behalf of particular users... the replies are probably intended specifically for those users. I'm running into something similar with emailed invitations from my site, and have been thinking about ways to get the replies back to the user doing the inviting.

You could try something like:

    From: "John Smith via ExampleCo" <replies+encoded-user-id@example.com>

where "encoded-user-id" is a signed and timestamped identifier that lets you identify your "John Smith" user, so you can forward the reply to them (or insert it in their newsfeed in your product, or whatever makes sense). You'd have to be very careful to validate incoming replies, to avoid creating an open mail relay or a vector for spammers to reach your users. (Services that implement anonymous/private replies, like Craigslist, use an approach like this.)

Does anyone know of any well-tested packages that safely provide this sort of reply forwarding? Or transactional ESPs that offer it directly?

[1]: http://stackoverflow.com/questions/32696850/is-the-reply-to-...

[2]: https://medium.com/@BraunDoug/windows-10-mail-client-broken-...

[3]: http://www.geekzone.co.nz/forums.asp?forumid=86&topicid=1949...

[4]: https://www.campaignmonitor.com/blog/email-marketing/2011/08...

[5]: https://www.mailjet.com/blog/the-noreply-dilemma-going-from-...

Thanks for the help! I'm glad to see I'm not the only one with these issues.

Thanks for the tips

I've had great luck with my personal email servers thanks to this tool: https://www.mail-tester.com/

Gives you a score and suggestions on improving it to reduce the chance of hitting the spam filter.

Cool tool. I'll have to play around with that and see how it goes. May end up adding it to the article :)

I like this one from NIST: https://www.had-pilot.com/py/had.html

You just send an email to the address at the bottom and it replies with a bunch of information regarding its SPF, DKIM and/or DMARC.

For something more complete: https://glockapps.com/

This gives you (in the paid version) a "seed list" of emails at various ISPs, so you can see how your email does with gmail vs yahoo. This is one of the several services, but the one we're most happy with.

Just to throw this out, I'm the guy responsible for one of the "top 10" DMARC reporting engines on the Internet in terms of volume. If anyone would like to chat about DMARC in the real world or about the DMARC reports (that they are very likely receiving from my employer's domain name), feel free to reach out to me.

My email is my username at my username dot org.

Google's Postmaster tools always tell me 100% DKIM, 100% SPF but a crazy varying amount of DMARC success rate. Postmark's weekly report gives me 99.9% DMARC compliance. Do you have an explanation or theory? Our envelope-from often differs from header-From but none of those domains have a strict DMARC alignment policy.

I believe you can explain some common cases in Postmark's weekly digest like this (but would really appreciate confirmation from an actual specialist in the field):

1. "Trusted sources" (DMARC fully/partially aligned), DKIM pass, but SPF fail: a recipient has forwarded your fully-aligned email.

2. "Untrusted sources" (DMARC not aligned), DKIM fail, SPF fail: genuine spam, or email forwarding that also rewrites headers in way that breaks DKIM (like the recent Hotmail/Outlook.com forwarding problem).

3. "Untrusted sources", DKIM pass, SPF pass: properly signed and SPF'd, but your envelope-from domain doesn't match the header-From domain. If your DMARC policy is reject or quarantine, these messages won't get delivered.

One way to get case 3 is with a vendor sending on your behalf, where you've included their SPF in your own record (so SPF pass), but they sign DKIM and set envelope-from using their domain. The DKIM is valid for your vendor, so passes, but doesn't match the From, so DMARC is not aligned and fails.

For example, UserVoice has this problem if you're using a custom From address in your domain. And Gmail shows this type of message as "From <you> via <vendor>".

I can't say without more details, but if you can reach out to me privately, I'm more than happy to dig through logs on our mail servers (well, dig through our Splunk platform...) and tell you exactly what we're seeing from your domain.

I resent that IP block based filtering is done anymore. Legit people are using VPS to send email, and email is a fundamental decentralized protocol of the internet. It's really crappy that in this age of DKIM/SPF/DMARC, we can't do away with IP filtering.

Thanks for posting this. I know it's on a WordPress site but definitely applies to way more people than just that audience. I hope to build the article out even more and get super specific with various vendors like Mandrill, Amazon SES, Sendgrid, etc.

I think it's pretty weak they're advertising the use of "~all" in their spf records. Either use "-all" or just don't use SPF I would say. If you can't make a decisive statement about your own domain then it won't be actionable for receivers that evaluate your records.

That's a totally fair point. I'm not entirely sure why most third parties are still using ~ in their documentation but it still seems to be the norm. I do like the definitive nature of -all.

IIRC ~all is the recommendation because hotmail/live told people to use ~all to prevent hardfails when hotmail's lookups timed out or if a particular mailserver IP was inaccessible during spam checks.

~all will result in your email being bounced around until accepted even if the IP doesn't match DNS records (more or less).

-all will result in hardfail if rejected by any TO mailserver.

-all isn't necessarily a slam dunk because it interacts badly with lots of mailing lists. There is standards work underway to improve this, but it's still an issue, for example https://www.ietf.org/mail-archive/web/ietf/current/msg87153....

If you use DMARC with a reject or quarantine policy, SPF hardfail ("-all") can prevent recipients from successfully forwarding mail you've sent them.

Some best practices for DKIM, SPF, and DMARC (as of mid-2015) in [1], including this:

> ...when an organization publishes p=reject [in DMARC], they should simultaneously change their SPF hard fail to SPF soft fail. ... A message that passes SPF and is forwarded will fail SPF. If a message hard fails SPF it will probably be marked as spam but if it soft fails, it will most likely still be accepted by the recipient. This forwarding failure possibility is why most organizations publish a soft fail record.

[1]: https://blogs.msdn.microsoft.com/tzink/2015/07/12/what-is-th...

Interesting note about DMARC, but still, if you're concerned about breaking forwarding for your domain, then why bother using spf at all? I still don't see the benefit of setting up ~all rules.

I believe that DMARC requires SPF. Since I want DMARC, I need to provide a compatible SPF, which means ~all.

(And I do want to implement DMARC. Not so much to improve deliverability of my own email, but rather to prevent delivery of malicious email pretending to be from my domain.)

Ok, just wondering, would "v=spf1 ?all" be the same in that case? I.e. a neutral spf policy?

Google specifically says[0]:

Create a TXT record containing this text: v=spf1 include:_spf.google.com ~all

Publishing an SPF record that uses -all instead of ~all may result in delivery problems. See Google IP address ranges for details about the addresses for the Google Apps mail servers.

[0]: https://support.google.com/a/answer/178723?hl=en

I host my own email at home on a domestic connection with a domestic fixed ipv4 and I only set up SPF.

Most e-mail providers accept my email.

More accurately: in the last two years only gmx.de rejecte one email.

I have found that the site https://mxtoolbox.com/ is very good for scanning your records for SPF/DMARC/DKIM and pointing out problems. Unfortunately even with implementing all of this some email providers are heavy handed with rejecting mail from smaller mail servers. For instance Yahoo will usually block all of your mail without appealing to them, and Verizon will not allow any email originating from a VPS source like Digital Ocean or AWS. Ironically I found that out when going to report someone on Verizon's network trying to brute force my SMTP server. Mail to abuse was rejected.

Sendgrid has a nice implementation for DKIM too. They setup 2 DKIM CNAME records that point to a DNS entry with a id number in it for you. Periodically, they rotate DKIM keys and by having the 2 CNAME's they can easily transition without concerns about something getting lost in transit.

  _domainkey.yoursite.com TXT "t=y; o=~;"

Does anyone know how necessary this entry is, as opposed to just having records beginning with selectors?

It seems like the t=y means that testing is on and to not actually block messages that fail DKIM, and o=~ means that some messages aren't signed. I'm not sure why the article is suggesting people use those settings, since they are entirely variable between different users and their config.

Say I want a personal domain and have the MX not be Google or Fastmail, how complete is the implementation of these standards on major email providers' SMTP setups? Would I need to do thorough research or is it reasonable to expect it to just work in a, say, European SMTP hoster's configuration?

Any major ISP with clue is at least checking SPF, DMARC and DKIM scores on its incoming edge smtpd that talks to the world. How they use the scoring and results for spam filtering varies widely.

I don't like the implication that the work outlined in this article is reasonable.

Imagine if there were a detailed guide on how to keep the post office from throwing out the letters you send?

Because mail that you personally send out by definition isn't spam - so you are doing work to get around broken spam filters.

why can't you just pay $10 or something as a deposit and, since you're not actually a spammer and nobody will ever actually mark what you send as spam, never lose that deposit.

This guide should be like four lines long and take 5 mi utes to follow.

I mean after glancing at that write-up, I'd never dream of running my own mail server. I use gmail. Why would I jump through hoops and still risk having letters I took the time to write, still marked as spam? I lose on two counts! (invest time, for a worse outcome.)

This part of the industry is broken. I think a deposit paid by non-spammers which they lose if people start marking their letters spam, might fix it.

I am surprised that almost no one seems to be using DKIM. I tried setting a higher spam weight on unsigned emails and half of my emails ended up in the spam folder.

SPF is used more frequently.

It's worth keeping in mind that the real value of SPF records isn't preventing you from receiving spam (aside from backscatter), but to prevent Joe-jobs.

I wrote this a while ago. I have SPF, DMARC and DKIM all implemented on my mail domain and I still get put in the spam folder:

http://penguindreams.org/blog/how-google-and-microsoft-made-...

I think part of it might be that I use Linode, and there are other spammers in their data centre, so I could just be on a subnet bad list. But I think a lot of it has to do with Google/Microsoft's spam filters just being crazy over aggressive.

Yeah, no doubt that a sketchy IP will basically negate any kind of authentication you have in place. Off to read your article...

Have you also set up reverse DNS and done it all for IPv6 too? The second part was what I was missing a while back. I do agree that Google and Microsoft are extremely strict with what they accept and it's not always easy to tell why you get thrown to the spam pile.

Yep, I think there was a post about setting up IPv6 correctly on HN a while back. Alternatively, you can make your host not respond to IPv6 at all.

If I remember correctly, the problem isn't that IPv6 needs to be set up, but that if your host listens on IPv6, Google will default to that and then your SPF/DMARC/DKIM setup needs to work with IPv6. If you don't have an external IPv6 address, then you don't need to configure SPF etc... for IPv6 and you should be fine.

It depends, really. When you're just starting out with sending email from a fresh domain and IP address, you will most likely be flagged no matter what. Do not underestimate signals that come from user interaction (e.g. open and click rates).

Deliverability is a reputation game.

A huge part of the problem is that while your personal smtpd might be set up impeccably, you're in the same /24 as a bunch of other low cost bulk hosting customers that have in the past several years set up VPS with much less clue than you. As a result the entire ARIN netblock that your server resides in has a "poor" IP space reputation as seen by the opaque incoming anti-spam measures put in place by Microsoft and Google.

There's very little that can be easily done about this other than moving your smtpd to an ipv4 address with an ISP that has never had an outgoing spam problem (such as for example a /24 that's been held by the same company for 8+ years, in ARIN/RIPE/APNIC/whatever space very tightly controlled by the network engineering team of a clueful local ISP where you know the staff).

There's a pretty direct inverse correlation between the cost of a hosting service ($5/mo VPS vs. minimum $200/mo colocation of a 1U server) and how much outgoing abuse traffic has been sent from the particular netblock assigned to the enduser customers. Cheap hosting company = poor IP space reputation.

And this is how the decentralized internet dies...

Yes, sort of, but not really a new problem. Low budget hosting companies have had poor IP space reputations and outgoing spam problems since such a thing as a hosting company began to exist, 20 years ago, way before there was such a thing as an x86-64 bare metal hypervisor or a VPS. The lower the cost the less clueful the customers. The problem is more on the google and MS side.

Not really. If you move to Mogadishu you're not going to be able to get a home equity loan. Choosing good neighbors is important online or offline.

>There's a pretty direct inverse correlation between the cost of a hosting service ($5/mo VPS vs. minimum $200/mo colocation of a 1U server) and how much outgoing abuse traffic has been sent from the particular netblock assigned to the enduser customers. Cheap hosting company = poor IP space reputation.

Eh...this is pretty much bullshit. Unless you're running your own, good-reputation AS (or happen to know someone running a clean AS who can either lend you an IP or announce a clean block for you), the /25|/29 assigned to you by your colo provider has just as good a chance as being dirty as that VPS providers.

Your colo provider is also more than likely happily delegating blocks to one of those cheap VPS providers out of the same larger netblock your IP. Chances are even good that you'll be given a /26 or a /29 that used to be used by one of the VPS providers.

If you see a provider offering "up to 256 clean IPs!!!", those aren't clean IPs. Those are, at best, greylisted IPs bought on the cheap and at worst /22's rented from poor rep /16's and rebadged in the hope no one will notice

Not really bullshit if you find the right hosting company. There's shitty $200/mo single server colo options, and less shitty ones. In my experience it comes down to personally knowing the people who run the ISP and how serious they are about network abuse issues (and general network engineering best practices). If you find a colo/cabinet space hosting company in a major city that's one of the founding members of a major IX there's somewhat more chance that they'll have network engineering staff who take such issues seriously.

On the customer side, one of the major bars to entry for clueless/spamming customers is whether it's possible to directly purchase hosting services online with a credit card and have them immediately provisioned and available. If you can get a VPS in 5 minutes by paypal it's easy to be clueless. If you need to set up and ship a server with its rails to a colo it's likely but not guaranteed that you have more clue than usual.

Yes, it's likely you might get an IP in a /26 that's part of a hosting company's much larger /22 or /20 which also contains shitty VPS. The key part there is to find a hosting company/ISP that doesn't do low budget hosting and never has.

Most people aren't in your or my position, where we might know (sometimes personally) the management at an ISP. Most people aren't in a position to build their own server. Heck, the vast majority of people who want to host any thing aren't in that position. And who wants to spend $200+/m to on their personal email?

The number of colo providers who refuse to allow customers like VPS hosting companies (who buy rooms not just rent a piddly little 2U from a shared cabinet), is pretty small.

All in, your advice isn't tenable for the majority of technophiles, let alone your slightly-more-technical-than-average user who wants to setup a mailserver. The barrier to entry, your way, is insurmountable.

Maybe it's time for a startup in the "DIY mailserver" space, that offers clean IPs for personal use mailservers and hand-holds through the process?

I agree that the vast majority of people who use email in general, or even the vast majority of people who own a domain name and want to have their own email server are not in our position...

The venn diagram of people who are capable of operating a secure Linux or *BSD based email server implementing, for example, SPF, DKIM and DMARC with postfix+opendkim+spamassassin+dovecot overlaps a great deal with the sort of people who want to colocate a $100-200/month server. A lot even have "free" colocation through their work at an ISP or with friends that have extra rack space and power for a small 1U box.

It certainly doesn't make sense to spend $200/mo on colocating a server just for email - but if you're colocating a whole physical server in this era, it's not hard to make it a box with 64GB or more of RAM and two good quality 512GB SSDs in RAID-1: Make it a hypervisor platform and put twenty of your own VPS on it doing many different things. Balanced with the need to not centralized too much stuff on one piece of hardware as a single point of failure. Public IP space availability depending, of course.

In my experience the best colo is with ISPs that are not actually colo companies, but will only do it as a side thing for people they know and trust. The absolute best colo I've ever had is with a company that has a core business doing X.509/SSL stuff for healthcare enterprise customers. Gear in racks two hops network-topology away from their core routers at a major IX point.

The vast majority of people who do not want to maintain and secure a world-facing Linux/BSD based smtpd are probably better off going with a google apps or hosted email solution where all of the smtp and imap/TLS1.2 infrastructure is handled for them.

I'd disagree, there are several packages that make it relatively beginner friendly (Mailinabox, sovereign) -- everyone has to learn some how. There are also lots of good tutorials, and one of the great pleasures of setting up a mailserver is once you've done it you can reuse the configs. It is a great learning experience for relative novices and "greybeards" alike.

You've just cut the number of people who can do it your way down to <1000 people (or may as well). No normal person, who wants to learn how to setup a mailserver and be successful at it, is going to drop ~$5k on a server or know people who run T1/T2 datacenters and are willing to pop you in a cage for free.

Your last sentences dismisses everyone who doesn't do things to your impossible standards as a waste.

You're basically saying no one should run a mailserver, or learn how, or be given the opportunity to learn how. Which is sad.

More people should learn how to run a mailserver, if only to better make decisions about when to outsource and when to keep inhouse.

I'm running on Linode and have been doing OK. Initially I was getting dropped in spam inboxes, but after getting things sorted out using mail tester, it seems to be better.

What is your score on here?: https://www.mail-tester.com/

Reply on news.ycombinator.com