Email Authenticity 101: DKIM, Dmarc, and SPF
alexblackie.com
alexblackie.com
I'd like to add that this is because with no MX record the MTA may still attempt to deliver email to the IP from the domain A/AAAA record.
This is a lesser known feature of SMTP, it is specified in rfc5321 section 5:
If an empty list of MXs is returned,
the address is treated as if it was associated with an implicit MX
RR, with a preference of 0, pointing to that host.
"[T]he SMTP client first looks up a DNS MX RR and if that is not found it falls back to looking up a DNS A or AAAA RR.
Many domains do not accept email, but do have A or AAAA records. If they have no MX records, senders will attempt to deliver mail to those A or AAAA records. "
Nothing like a good opt-out to make you feel like you're signing up to a Do Not Call list.
I guess we can't blame email too much, it's an incredibly old protocol. But at some point, some quirks have got to be worth rethinking, right?
Maybe we can deprecate the fallback from MX to A/AAAA for the next, oh I don't know, two hundred years before we can move on.
Most of my domains don't send mail, however I do prefer to receive mail to the standard aliases (webmaster@, abuse@, etc) and often a catchall. Even for domains I've got parked and awaiting deployment.
Am I thinking too much, not enough, or is the opening phrase of the advice meant to be "Another tip for domains that don't intend to receive email"?
At some point I just accepted reality and paid for a third party email server (Runbox) but the situation with the monopoly of these giants marking as spam anything they don't manage made just really sad.
Email is not this wonderful, democratic system of equals. Email is more like a school playground where a handful of enormous bullies control everything. You either learn to work with them, cater to their demands, or you'll never be able to visit large sections of the playground.
Gmail is relatively easy to appease, mostly just configuring your nameservers and email headers to Google's standards, then slowly building "reputation". Hotmail will block an entire domain (IP?) randomly every few years, then you fill out one of their infamous forms and pray they unblock you quickly. Yahoo (technically Verizon now) is even more difficult to work with, they block you out of nowhere and I've never figured out how to appeal. Seems like they do some kind of ratelimiting for smaller email servers, I don't know if it's some kind of punishment or just by default.
Then there's the dozen other major email providers. Some easy to work with, others not so much.
Do you mean SPF, DKIM and DMARC or something else? I will check the records again, just in case, but afaik they are perfect and Gmail still often classifies my mail as spam. It is annoying as hell (and your comparison to playground bullies is very apt).
It's not a very good system.
Ah. But the million dollar question is WHERE did you run that server?
If you ran it off your home connection then there's your problem.
Pretty much everyone on the receiving end (ISPs, corporates etc.) will either block or assign a low reputation to home ISP IP ranges.
So basically you can do all the DKIM and SPF you like on your home MX server, but you're going to have the high hurdle of your untrustworthy IP range to fix first !
EDIT: Just saw you said "cheap Scaleway" ... well that's a similar sort of problem. Perceived "spammer friendly" hosts tend to be assigned a low rep in spam scanning. You won't be auto-blocked like residential IPs are, just a low rep set. (for any lawyers reading, I am NOT making any comment on Scaleway here, I am speaking in general terms !)
Like you, I have properly set up DKIM and SPF and blah blah.
I also have a 12 year "clean" IP from he.net.
Sometimes gmail will mark something I send as spam that is, like yours, a reply to their user and is a recipient that I have a years long history of corresponding with.
That's frustrating but what really makes me angry is that there is no indication given that this occurs. Why can't google send a proper bounce or warning message ... or even an error code that I could at least see in my logs ?
In the arms race between spammers and (providers) are error indications exploited in some way ?
Or is google just being lazy ?
Relevant providers usually just require a few days of sending legit mail and occasionally asking to be unblocked, but there seems to be no chance getting into the inbox on GMail and Outlook with a "too low" sending volume.
If it wasn't for regulations, I would have already given up and switched to GMail.
https://jschumacher.info/2021/05/running-a-private-mail-serv...
That was a good article, and your site seems to have a lot of useful information. Does it have an RSS feed?
rspamd is worth checking out as an alternative to SA if you need something more advanced than procmail.
For my personal e-mail most of the time I am just on receiving end. For receiving e-mail I don't see much problem. For receiving email only you don't need to setup any of the DKIM/SPF/DMARC stuff.
Occasionally I send some e-mail out but you are not going to be marked as spammer if you sent 1 e-mail a week from your IP address. Most likely it will be delivered and accepted by whatever provider receiving person is using, might be marked as spam - but well you probably can call up that person and ask them to check spam folder.
For spam - don't put your email into every web page that asks you to do it and you should be good. My gmail account that is now ~20 years old started to receive loads of spam only last year when email lists from various leaks started to go around internet. My other email accounts that get loads of spam are on shitty providers that send their own spam.
I'm similar to your boat, so this is just one more reason I'm going to give it a go, I think. It'll take time to move the stuff I care about off Gmail anyway, so I can sort out the spam stuff as I go. And I'd really like the flexibility of controlling my mail in ways I can't really right now with a hosted provider anyway.
For a time i DID host my own emails. Out of my house, with a business line and static IP.
Google Apps was free at the time and I registered my domain and over time that has become the easiest (its still free for me because i grandfathered in back in like...2014.)
With that said, more and more I want my own privacy and control. And VPS services are cheap. Im leaning toward moving to Mail in a Box or something similar to what OP talked about in his blog. Its really just me and my wife so not much volume. Could probably host it on Linode no problem.
https://vadosware.io/post/setting-up-ses-with-pulumi
I'm a big fan of Pulumi and have used this setup for 2 projects so far (it's part of a bigger framework of repeatable infra for SaaS projects I've been hacking on), and I can say it definitely delivers (pun intended).
Code in the blog post might be a little stale -- I've just put an update in there to note that SES recently changed to using DKIM records primarily for domain verification, so the aws.ses.DomainIdentityVerification (Pulumi object) is no longer needed. One of the nice things about using Pulumi to do this is that since AWS allows you to use pre-made DKIM keys now, I read one right off disk (look into how to make a dkim key[0], it's interesting), and use it for both my personal email (I run maddy[1]) tied to the domain and SES.
[0]: https://lxadm.com/Generating_DKIM_key_with_openssl
[1]: https://maddy.email
Certificates are certificates are certificates.
They're 1KB files full of random numbers. What you use them for shouldn't alter their price... which ought to be zero.
The facts that a handful of near-monopolistic CAs have managed to control protocols, set standards, and generally funnel money into their coffers that need not have been spent at all is one of the worst examples of naked capitalistic greed I have ever seen.
Rent seeking, pure and simple.
> Thousands and thousands of dollars for a BIMI certificate is just ludicrous.
VMCs are between 800 and 1500 USD
> Certificates are certificates are certificates.
The x509 certificate is, but the process to verify your ownership of the supplied 'mark' (your logo) is not.
The VMC issuing procedure is very involved, it requires the CA to confirm with your local trademark office that your company owns the trademark. For VMC the CA is also required to verify (by phone) that the person who requested the VMC does indeed work for the company, they are also required to verify that the person who requested to certificate is allowed by the organization to do so.
The CAs are actually doing work here, there are real costs.
FWIW: Digicert currently sells VMC for a discounted price of 800USD, which they claim is not even economically viable. Without the discount expect VMCs to be ~1500USD.
Your argument is essentially identical to the arguments usually made to explain why EV certificates are expensive, and nobody is buying those.
And then clicking 'renew' once per year?
How could you possibly charge less than a thousand dollars for that...
Again, the 'renew' click is not why this is expensive. It's just a cryptographic function that signs a bunch of data, Lets Encrypt has long proven that certificates can be created free-of-charge. However, having a human verifying that everything checks out is the expensive part. Having that human work in an environment, following procedures that passes public audits is expensive. None of the trademark offices is going to do your trademark validations for free. No-one is going to staff a call-centre for free.
You are right about the technical part of creating a certificate being trivially easy, but I believe you truly underestimate the costs of running a CA that is capable of delivering VMCs.
Maybe that some company can do it for less than the current prices, I don't know. Competition will show that eventually. But if you truly think that you can do it for less than the current CAs, then start a CA yourself and start competing.
So you provide the CA with an trademark ID number that they can look up and they verify that you represent the company that owns the trademark. It's like 10 minutes on top of the existing EV process, but it's more than double the price.
Per domain.
One of my customers has dozens of domain names!
The domain names are in the AN section of the cert. You can have as many as you want in there (as long as they share the same logo)
Please be aware that the BIMI standard is far from finished. There are some serious documentation ambiguities that pose a risk to how well BIMI might function, and how email providers will adopt the technology. Obviously, the CAs are now pushing hard for BIMI, since VMCs are a new source of recurring business for them.
In my personal opinion I think that BIMI is pushed to early, which could leave brands attempting to adopt BIMI disappointed. Obtaining a VMC is quite involved (and expensive) for small businesses, and the potential benefits are (currently) quite small, as not many email service providers support BIMI.
Don't get me wrong though, I think BIMI has a great goal of pushing the DMARC adoption rate (remember, you must implement strict DMARC for BIMI to work). I just don't see BIMI offering any reliable brand exposure for the coming years. Only time will tell I guess.
What does this provide in contrast to DKIM? Perhaps there are logistical problems with DKIM, but I wouldn't know. I wonder why it isn't more widely spread in the first place. The receiver can determine the requirement, which is the best solution imho.
For Google it seems to be more of a marketing issue than security. Furthermore provider could track domain requests. That isn't just suspicious, this is almost malicious. There is a reason mail clients don't display pictures outright.
I hope my mail client will never support this.
It does pass the checks though [1], but does not really work because of the missing certificate which you can for example get at DigiCert [2] once your trademark has been approved..
The domains I configured it for are: antwise.com and vimalin.com so you can see at [1] how it looks like when it is supposed to work.
[1] https://bimigroup.org/bimi-generator/
[2] https://www.digicert.com/tls-ssl/verified-mark-certificates
While it‘s certainly true, to me it‘s missing the most important aspect for practical use here: Email must be deliverable.
I only started down the rabbit hole of email authentication long after we discovered important client and outreach mail never reached its intended sender.
Back then, most of this stuff was hidden in some layers of documentation in G Suite instead of in your face after setting up your custom domain (where it would belong imho).
If you‘re setting up your freshly bought startup domain, please do yourself a favor and follow this or a similar guide from A-Z.
1. Don't use strict (adkim=s; aspf=s) if you expect your emails to be forwarded (https://www.dmarcanalyzer.com/forwarding-within-dmarc/).
2. Microsoft (Outlook/Office365) ignores DMARC p=reject by default (https://docs.microsoft.com/en-us/microsoft-365/security/offi...)
Neither of this settings have anything to do with forwarding.
aspf and adkim are for SPF/DKIM alignment, where 'r' (relaxed) allows subdomains, and 's' (strict) does not.
aspf=s requires RFC5321.MailFrom and RFC5322.From to match exactly (no subdomain difference) for the SPF to be qualified as aligned.
adkim=s requires the DKIM key to be published under the same subdomain as found in the RFC5322.From address for the DKIM to be qualified as aligned.
Forwarding will break SPF in most cases anyway, this is unrelated to your aspf setting (unless you are forwarding between your own subdomains).
DKIM should not break by forwarding, regardless of your adkim setting, because forwarding does (should) not rewrite the RFC5322.From address.
For more explanation, see here: https://www.mailhardener.com/kb/dmarc#relaxed-vs-strict-mode
> 2. Microsoft (Outlook/Office365) ignores DMARC p=reject by default (https://docs.microsoft.com/en-us/microsoft-365/security/offi...)
They don't ignore it, they treat it similar to p=quarantine. So there still is a difference between p=none (or no DMARC at all) and p=reject.
I don't think so.
SPF attests that the mail was sent by an IP address that is authorized to send for a certain domain.
DKIM attests that various headers, usually including the From: address, are approved by the owner of the domain.
SPF prevents random people on the internet sending mail that purports to come from your domain. DKIM prevents people inserting forged headers into emails. These goals are orthogonal; neither is a stronger signal than the other.
SPF has many issues. One of which is that including external email providers will include large blocks of IP addresses to your SPF that you don't control. IP spoofing is also a thing. SPF also breaks horribly with forwarding email services (like catch-all inboxes). The SPF lookup limit of 10 additional lookups is way too low for most modern email setups.
SPF is just not very robust, and badly understood by most people (leading to many configuration mistakes). Hence that receivers cannot get much trust from the SPF outcome alone.
DKIM does send a "stronger signal" in the sense that it is more robust than SPF, and contains cryptographic proof. He who owns the private key is trusted by the domain (who published the public key) to send email on behalf of the domain. DKIM also does not suffer from the forwarding issue that SPF has.
As a result, a valid DKIM signature often does result in a higher trust rating by the receiver, than just an SPF pass.
DKIM + DMARC enforcement is the most reliable way to go. Obviously you still want to setup SPF as good as you can (until you start hitting size and lookup limits).
Disclaimer: I'm the founder of Mailhardener, an e-mail hardening platform.
IP spoofing is pretty hard to do with 2-way communication channels like TCP and a lot harder with TLS too. usually IP spoofing only works meaningfully with UDP and protocols that allow for transactions were you don't need replies from the target. to pull this of you would need to do BGP hijacking in order to receive return traffic which is a little above the efforts spammers will make.
Are you saying that my core point is wrong, that SPF and DKIM serve orthogonal purposes, and that it's incorrect that one is somehow "more robust" than the other?
I can't see what I said that you disagree with, except that somehow "DKIM and DMARC enforcement is the most reliable way to go". You've just said that you disagree, without providing much evidence.
You need DKIM if you want deliverability to (principally) GMail. You need SPF if you don't want your mailserver blacklisted. You don't need DMARC, really, unless you send so many emails that you need some kind of automated reporting infrastructure.
Yes, but SPF ties the IP address only to the Sender address (MAIL FROM) - so utterly useless in stopping phishing emails for example, if attackers can use said domain in From address.
> DKIM attests that various headers, usually including the From: address, are approved by the owner of the domain.
As the From address (and implicitly domain) is the one people see (and trust) DKIM in a way should matter more.
Also, misconfigured SPF records happen from time to time (on big domains actually), where bad actors can then take advantage of them and abuse delivery, much harder for a DKIM private key to end up in the wrong hands.
I suppose technically there is the problem that a domain with a good reputation can be hacked or bought, but the costs of that happening should be orders of magnitude higher than the cost of buying a new domain.
Requiring new mail-serving domains to post a bond for good behaviour seems like it would similarly increase the costs of buying a good reputation, but the problem is coming up with a fair process to decide when a bond should be forfeited.
I'd be reluctant to give the ITU any control over the global email network, so instead perhaps there could be some system for countries to delegate the bond-forfeiting power to companies of their choosing, with each country having a vote in proportion to the cube root of their population size. All existing domains would be grandfathered in to not need bonds, so the potential damage from abuse of this system should be minimal.
* I'm thinking something similar to SSL: write email, create hash (one hash per recipient), submit signing request for hash, CA signs request and returns stamp (certificate), stamp is transmitted along with email. Additionally, allow people to give each other (or sites/newsletters they would like to receive) 'rubber stamps', allowing them to create (free) stamps for emails addressed to each other... And since no idea of mine is ever original I'm certain several smarter people have already written multiple RFC's on this and the idea never took off because of something I'm overlooking at the moment. Ah well.
And mailing list operators, and charities, and free software projects, and...
To be fair, your "rubber stamps" idea does mitigate this problem a lot, but I think the issue still remains that it's hard to get email providers to agree to drop emails from senders that haven't upgraded to this new system, as well as agreeing on a globally consistent set of CAs who are trusted to not act as a cartel.
It's even more difficult to get all existing pieces of email sending infrastructure to simultaneously upgrade to support users entering their rubber stamps, not to mention the security concerns of dealing with the funds needed to buy the non-rubber stamps. Perhaps the big email providers could force everyone to implement this as of a certain flag day, but I imagine the response to this weird new tax would be less than enthusiastic.
You will whitelist these once you subscribe to them.
There’s a number of things I like to kick the tires on without giving my actual email. So I’ll use a temp email and more times than not it’s detected to be as such. How is that known?
1. The server accepts emails to all emails addresses. If I want to check if a service is a temporary email service or not, hit a few random addresses. If they're accepted and do not bounce that's definitley a little suspect.
2. known addresses If I were going to prevent temp emails, I'd look out for temp-mail type domains. This is a game of cat and mouse though, but blocking the known ones is probably enough to wipe out 80% of these accounts.
3. Low open rate Speaks for its self, especially with disposable emails.
4. random addresses Usually emails are `first.last`, so getting f00b4rb4z@temporaryemailsite stands out
5. Your smtp server IPs are made available via your MX records Offer (paid?) users a new domain to point their domains to with an IP which won't have been blacklisted by providers.
Ways of defending against boil down to: bounce addresses that haven't yet been created, allow users to use their own domains, offer random names rather than random strings as addresses. These are all areas I'm working on over time, but with all side projects it takes time :p
[-] Outlook rejected my emails: vague reason given, but the fact is, Outlook rejects entire IP address allocations (for example, DigialOcean's entire VPS network), even individual DigitalOcean IPs which aren't publicly blacklisted. There is a proper process in-place for small upstarts like myself to get de-listed, but its not a given right that the process will favor the applicant.
[-] Gmail. Email received. No problem, no spam label. And I'm quite thankful to Google/Gmail for accepting my mail based on metrics other than IP addresses alone.
So while I can't answer why they're allowing obvious spam through their filter, I can say that at the present moment they are not as ruthless in blocking /16 networks (or whatever size) as some providers based on IP addresses alone (e.g., Outlook, E365).
Lack of verbose feedback / delivery errors makes debugging issues in your own really hard, so you might want to contact someone who knows a thing or two about emails for help.
You say domains
To ensure we're speaking to, not past, each other it will be essential to clarify one thing: what do you mean precisely by "setting up new domains", "cheap/ free domains"?
Domains, I'm not interested in chatting about those, but IP address space, then yes I'm all ears. If you can reliably setup new, fully self-hosted mail servers (i.e., not using SendGrid, MailGun etc), with port 25 open for both send and receive, on cheap, accessible VPS servers...
And...
Send email messages from those new servers directly to your Outlook's inbox.
Then you Good Sir, have the secret sauce.
*Caveat: AWS does have a lot of un-blacklisted IP address space, however, port 25 remains blocked for new customers without a "valid use-case" and an unspecified amount of payment history. Similar for Azure. So if your methodology is built around AWS/Azure then kudos, but for new-comers, it would be unrealistic from my experience to suggest the path forward will be without major hurdles and/or delays.
The lack of verbose feedback in the rejection reason from Outlook was, as I see it, a feature not a bug. A bit of friction to turn away the undetermined ones. I followed the trail of clues from the top search results of the error message and eventually got my new IP unblocked through the proper channel. So... I have made it from bottom tier (outright rejection of emails), to 2nd-bottom tier (directly to spam/junk folder). Next: build domain/ip reputation ever so slowly.
There's no secret sauce but there is a myriad of ways to shoot yourself in the foot when setting up a mail server from scratch. E.g. many standard setups may try delivering email over IPv6 if available which currently is a terrible idea but might be completely opaque to the user. Obviously, the results will also depend on what is it that you're sending as everyone uses point-based (or even ML) systems nowadays and headers + contents can earn you either negative or positive points.
Would be interested to hear from others who've acquired new DO IPs that weren't blocked on that list though. Change my view of the whole scenario.
What's more, though, is that there's actually a lot of false-positives. Even emails with people I've talked to in the past. Probably at least one false positive per week.
And my spam folder contains almost none actual spam letters, they just stop arriving a few years ago dispite my email address is very public.
DKIM, Dmarc and SPF are not an end user burden.
Additionally, PGP relies on a "web of trust" managed directly by the user. There is no way for a server to automatically discover the key from $YOURBANK to decide if the email is legit; you need the bank to give you their fingerprint through a side-channel. _pka DNS records help, but is that simpler than SPF?
Why not? To check a signature you're using a public key of the other party, so nothing prevents you from adding a milter that would check incoming mail against a list of public keys you trust, without compromising/sharing any secret key material with the server. (As long as you trust your server.)
Anyway, point of PGP is that you can do the checks end to end between MUAs, not just between MTAs. Relying on DNS to distribute key material is not that great anyway, so PGP with keys shared via a better channel, is more secure. (Less convenient too, perhaps, depending on your MUA.)
It's much less stressing to work with these when you don't yet have anything important going on.
Cue that email solutions checklist meme.
but then it makes email so environmentally unfriendly, may as well just send dead trees.
edit: interesting, didn't know the idea of arbitrarily expensive computations as adding costs to combat abuse was first presented in the early 90s: https://en.wikipedia.org/wiki/Cynthia_Dwork
If I remember correctly his proposal was to have the sender solve a cryptographic puzzle, which takes a bit of CPU time. For normal every day use you wouldn't notice, but this subtle costs and throughput limit would make spamming no longer economically viable.
But then I googled around:
https://serverfault.com/questions/611575/will-mailing-lists-...
It seems that SPF, DKIM and DMARC can all cause failure wrt. mailing lists depending on how messages are rewritten.
I can see that argument, but - it's kinda a philosophical question about "who the sender is". Is the the person who typed out the text? or is it the server which transcribed that text into N new emails?
The ML server will verify the original authors SPF. The N recipients will verify the ML servers SPF - the chain (which matches the series of MTA's involved) is still verified end to end.
> It'll break when used together with DMARC's alignment checks.
Yea, DMARC is a much bigger issue for mailing lists, but that's no reason to say "A mailing list will _always_ break SPF" - a well configured* ML has no issues with SPF at all.
* And, yes - the definition of "well configured" had to change when SPF was introduced, that's of course annoying, but there has been many many years for ML operators to make these changes.
The recipients have no way to check that the mailing list server has checked SPF/DKIM/DMARC. Mailing lists very rarely drop messages because of a failing SPF/DKIM/DMARC check.
ARC tries to fix this, but requires recipients to trust the mailing list server. Just using plain DKIM is much better, recipients can just treat ML-forwarded emails just like direct emails.
Both fill in gaps in the use case of the other.
DKIM completely supersedes SPF, hence my recommendation to just skip SPF.
Additionally, DKIM keys need to be rotated periodically just like SSL. Many services like Sendgrid or ProtonMail will handle this for you now by setting up multiple CNAME records so they can rotate the keys for you, but it only works with that sender.
SPF helps to address the gaps. I totally agree that strict DMARC policy plus DKIM should be enough though.
> Authenticated Received Chain (ARC) is an email authentication system designed to allow an intermediate mail server like a mailing list or forwarding service to sign an email's original authentication results. This allows a receiving service to validate an email when the email's SPF and DKIM records are rendered invalid by an intermediate server's processing.[1]
* https://en.wikipedia.org/wiki/Authenticated_Received_Chain