How to prevent email spoofing, using an unholy combination of silly standards
simonandrews.ca
simonandrews.ca
I've personally had a much harder time getting DNSSEC to work than I ever had issues with the various email DNS records. Even that is manageable if you're willing to Google around for a while.
My solution to getting self-hosted email right is using Mailcow. A cheap VPS (Contabo FTW) plus Docker is all you need to get it to work, and it spits out all the DNS records you might need to add. That also includes autoconfig records for tons of software, which is nice.
$1 - turn the screw
$9,999 - knowing which screw to turn
If you're thinking of setting up a program, it's worth your time to read over some existing program policies - those policies encapsulate a lot of experience that the existing programs have had on their platforms. Your ideal policy will probably differ, but it's worth thinking about why the other program policies are the way they are.
It's nontrivial to move away from e-mail-as-a-service. It shouldn't be, but we're stuck with decades of legacy and abuse. Years ago there was a statistic doing the rounds that either 80% or 99% of e-mail traffic was spam; I'm confident that is still the case, but thanks to technologies and big providers, end-users only see a fraction of it ending up even in their spam box, and even less in their actual mail. There was one in my non-spam gmail inbox the other day, the first one in years.
I run a wordpress blog with comments enabled as well, Akismet has stopped a million spam comments (99% of comments attempted to be poasted).
They filter like 99.99% of the junk. Usually 200-1000 a week (had a low of 20 a few weeks ago). They even show me what they are filtering. That is just 1 account. I have toyed with the idea of setting up my own domain and having my family have all their own emails. But that would mean managing it. Not terribly hard but just one more thing I do not want to mess with. 30 years ago I would have done it. Now I would rather it 'just work'.
Managing spam is a pain. I have been using the 'unsubscribe' links recently. You say then you are just verifying it is real. Well yeah, not like they are going to stop anyway. Worth a shot, but many times they actually respect it, unlike 15 years ago.
As for configuring a mail server, yes, that's definitely way harder. But these days, most companies outsource that to the likes of Google or Microsoft, until they get large enough to justify administering their own. There's exceptions to every rule, etc. etc., but every company I've worked at has either used G Suite or Office 365.
The result being: many companies have email services running, but don't have anybody whose day-to-day is understanding how it works and how it's secured.
You're completely right, of course. In my experience, these generators usually explain what the generated code means, though. These records are a lot easier to read than they are to write if all you've got is a manual and a technical specification.
I haven't used any hosted mailing services myself, but I can't imagine their control panels don't have either an option to generate the necessarily policies for you or an extensive guide on how you should configure these records and why. These records are the only part of the mail ecosystem these hosted platforms can't manage (unless you also let them do DNS) so they're a crucial step of the onboarding process.
On generators, a lot of them left me in an uncanny valley. They were supposed to make everything easier, but didn't explain the basic concepts, so I didn't know the values to even put into the generator. After I nailed down some of the basics, the generators just started making sense.
YMMV. There's an infinite combination of mail hosts/servers, DNS providers, and record generators. Collating all that information the first time can be overwhelming, if DNS or email aren't an area of expertise, but I'm sure there's a low-friction path out there. A Northwest Passage of email security. I'd love to find it.
Importantly you can also verify the correctness of the configuration, which isn’t the case for all anti-spam and anti-spoofing measures.
I have had to revisit these a few times for companies that change their setup standards and then dont tell their customers (same has happened to SPF but that is much easier to audit/fix).
https://www.reddit.com/r/sysadmin/comments/c5p7e8/dmarc_bloc...
If you're capable of running your own mail server then you know more about this stuff than most people who use e-mail do, and the article isn't for you.
I wonder if any other domain providers do this?
This isn't strictly true (that you can have as many as you want), because SPF has a (IMO) very silly hard limit of 10 DNS lookups per record. From RFC 7208:
> Some mechanisms and modifiers (collectively, "terms") cause DNS queries at the time of evaluation, and some do not. The following terms cause DNS queries: the "include", "a", "mx", "ptr", and "exists" mechanisms, and the "redirect" modifier. SPF implementations MUST limit the total number of those terms to 10 during SPF evaluation, to avoid unreasonable load on the DNS. If this limit is exceeded, the implementation MUST return "permerror".
We see this not infrequently at Fastmail, when customers report DMARC validation problems, and the answer turns out to be that they've got too many includes in their SPF records, so SPF always fails.
The only way this will help you is if you recursively de-reference all the records, and replace them with a list of allowed sender IPs. Now you have a new problem. You only get 255 characters in a single string in DNS. You can chain multiple 255 character strings together into a single record though, so you can get up to 4K characters. But, if you have a bunch of authorized spoofers (e.g., mailchimp) that different departments of your organization uses, even 4K characters may not be enough (source: I went through this exercise a couple years ago and our organization would have needed closer to 8K characters for the de-referenced records).
Our solution is to include our own outbound MTAs, and ?all. On inbound, we don't use SPF as a signal at all, as there are too many sites with misconfigured records.
Long term, we are trying to get each department to send spoofed mail using per-department subdomains, e.g., info@marketing.example.org, data@research.example.org, etc.
DKIM + DMARC is designed to handle the above without any of the issues SPF has.
I also typically have at least 2 for clients, including evidently google's 4. so this is important to know!
If your spf is too heavy, add universal spf and then watch it work by either sending a test message or rechecking your spf with https://vamsoft.com/support/tools/spf-policy-tester
Any domain can toss "v=spf1 include:UniversalSPF.org -include:x.UniversalSPF.org" in front of their broken SPF record to automagically clean and fix it. This authorizes mail that the domain owner expects to pass, and fails the mail they expect to fail.
Source: One of my startup's covid projects was creating and giving away https://UniversalSPF.org. We'd already been providing SPF Compression commercially since c. 2015.
It's free to use and already trusted by several hundred businesses.
Here's a good spf evaluator in case you want to see universal spf fix your domain's policy: https://vamsoft.com/support/tools/spf-policy-tester and a more technical deep dive for you other command line geeks: https://fraudmarc.com/introducing-universal-spf/
Based on a cursory inspection, it looks like UniversalSPF makes use of a custom DNS server and the the %{i} and %{o} macros for SPF.
Before this, I wasn't even aware that SPF had macros!
In this case configuration is handled by an unknown entity that you no contractual obligations to you. Don’t do it.
Here is a guide: https://www.gov.uk/guidance/protect-domains-that-dont-send-e...
I suppose that if you already know how to run your own mail server and are familiar enough with all the moving parts, sure, it's easy. But if you have a problem that needs to be solved, and you think the answer is running your own mail server, you now have two problems.
The "unholy combination of silly standards" is an excellent way of putting it, but it goes back to the 1970's. SMTP was originally based on FTP. Anonymous FTP, at that. It was designed to send text- nothing more. And really that's all it can do still. We can thank Base64 for binary attachments that are at 30% bigger than their unattached size.
Yeah, mail is a problem. And it always will be because it's a 50 year old technology that was designed for a collection of a few hundred servers ran by technologists who trusted each other. SPF, DKIM, and DMARC (and literally everything after except for "send text to so and so") is an afterthought.
That said, a technically inclined person would probably be able to set up a server in a day or three and understand the basics of how it all ties together.
Just because not that many people understand how to set it up doesn't make it hard. But it's definetly showing it's age and could be much simpler to use.
I've set up and run all of these, but that doesn't mean that it doesn't frustrate the heck out of me whenever I have to deal with them. A good, handy reference like this will be quite useful.
This is one thing I really enjoyed about setting up ProtonMail with my personal domain. They make it really simple to validate that SPF, DKIM, and DMARC are set up properly.
If Google Workspace/G Suite started pushing admins to use DMARC, and provided tools to make it easy, so many more domains would be protected, particularly in startup world.
Tutanota are just as straightforward, but they're both in the security game so not much of a surprise there.
Another confusion is the ARC, the acronym was alike so I though ARC has some relation to DMARC where they are totally different :).
I think to help new people understand them, we should stop group DMARC with SPF/DKIM. SPF and DKIM are the thing you set up and you control. But DMARC is just a policy to tell the recipient what they should do if something failed SPF or DKIM. And it's totally up to recipient's mail server to support and follow standard of DMARC.
ARC is a much better alternative that builds on DKIM and allows intermediate custodians of mail.
Many European countries are forcing DMARC adoption for government infrastructure and it works just fine.
And yet even Googles docs recommend ~all.
If you're using 3rd party bulk senders you probably want to consider the cost and benefit of -all vs ~all
Arguably there's nothing more clear than writing it manually over telnet. If you haven't done so yet it's kind of fun to try it out once.
I can't do anything about the attempted spoofs; I'm not going to track down everyone's open relays, and if I would, DMARC reports aren't really enough anyway.
At the time, Yahoo, Google and Microsoft all sent reports, which is a good portion of email, although certainly missing a lot. I think there were a few other smaller names, which I no longer remember.
Of course, it almost doesn't matter. Spam/phishing mail to our users still continued, they just stopped spoofing our address. It's not like very many people look at the domain mail claims to be from anyway; modern clients hide it, too.
This also means that emails from your domain is less likely to get marked as spam, which can be a significant win for smaller domains.
Without it, generally only partial validation of the envelope address happens by i.e. SPF.
I guess TFAuthor hasn't done a cursory google search, because if they had they'd have found the ISPmail tutorial[0] which is clear, concise, mostly correct, and approachable if you aren't a total novice to Linux.
SMTP is hopelessly naive by the standards of today as it was designed to work in a much smaller and more trusted environment. The additional reputation protocols are well intentioned and work well if the receiver acts on them and are not at all silly or unholy. They are not difficult to configure but neither are they as simple as a redesigned protocol might be.
If you don't need to setup your own mail servers or have a desire to develop and maintain skills in that area then use a pre-configured service.
Otherwise I am fairly sure that configuring these things is far simpler than most trade skills. I don't think there is any comparison between the time it would take to read some docs and add a few lines to a config file and learning to become a proficient welder. If you work with servers and want to be good at your craft I think email is still one of the services people should know how to configure and secure.
> 3. The Fastmail SMTP server generates a signature using the secret key, and attaches it to the email, along with a ‘selector’ (a string specifying which which of many possible secret keys key it used), then sends the email to example.com's receiving server.
> 4. Google email server receives this email. It's from sadl.io and has a fastmail DKIM ‘selector”, so it gets the DNS records for that selector in that domain.
> 5. The email has a signature and selector embedded, and the DNS records for the selector in the sadl.io domain declares a public DKIM keys which can be used to verify that signature.
> 6. That DKIM keys matches the one used to make this signature. So the DKIM test passes.
Crucially, there is no way to list all existing DKIM keys for a domain without knowing all the selector strings. Of course you could do a brute-force search and find a lot of them, but you could never know you found them all without doing a full zone transfer (or if DNSSEC is used with the old NSEC records, which allows enumeration).
If you just have a random Linux server that someone has run "apt-get install php" on, it probably hasn't been configured and so that command won't do anything. eg https://www.quackit.com/php/tutorial/php_mail_configuration....
Ps. Because you can't rely on PHP built-in email functions being configured properly and because apps generally want more control over how email is sent most modern apps will use their own library that makes SMTP connections itself. For example here is Symphony - note the first section on setting up transports: https://symfony.com/doc/current/mailer.html#using-built-in-t...
> SPF and DKIM are used as indicators of whether an email is spoofed or not. But if you added an SPF record on your domain, and you forget to add one of your email systems - say, Postmark, which you use to send mission-critical notifications from your application to your customers - then your customers could stop getting emails. If you added DKIM keys to your domain, but one of your email services doesn't support DKIM - or you forgot to add DKIM keys for that service - your customers could stop getting emails.
Setting up DKIM alone for one email service should still allow mails sent by any other email service (i.e. one not using DKIM) to be delivered. (That is, of course, if DMARC is not also set up with a strict policy.)
That is to say, DKIM is not that risky to set up and use, by itself.
DKIM = This specific email can by verified/authenticated by my DNS.
DMARC = What to do with email that doesn't comply with above. And how to tell me about it (if you want to).
SPF is basically useless due to the centralization of email providers. It was created to be locked to a company specific email server. E.g. Only accept mail from exchangeserver.company.com, and reject everything not from that server. Now everyone authorizes Microsoft or Google to send their mail.
I wouldn’t be surprised if there’s also a mail daemon that also did DNS for you. Everything in one simple MTA. Begone, Exim4 mega config and update-exim4.conf.conf.conf!
If it is really that important, businesses should be signing their emails.
Not so for email. It just stopped being worth it about 10 years ago or so. Regularly fine-tuning your configuration to not only not be marked as spam in originating mails, but to also filter incoming spam, walking the thin line between filtering too much and too little, was just no fun.
I looked out for an email provider that provides the flexibility that I'm used to (multiple domains, catchall for subdomains, sieve filtering etc.), pay them a few dollars per month or year, and haven't looked back. My DNS is still homegrown and I handwrite my zones, but the MX just points to my provider.
[1]: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme
I need to move my domains off of OVH...
Unfortunately, as much as I sympathize with this article, I can't pass this around to colleagues as technical reference, because it's just a rant. A rant with helpful information, but a rant nonetheless.
I need reference documentation that isn't going to disappear when the author decides to change blogging platforms, etc. So, below are the RFCs, with sections, that actually matter.
DKIM and DMARC are not required to have a functional piece of software that sends emails today, though. Just use SPF and you should be fine. A TXT record with @ for the current origin[1] or the relevant host with the following is sufficient[2]:
v=spf1 include:example.com ~all
If you're pointing to your own DNS records, though, don't use include. Use a. As far as I understand it: include[3] is others, or as RFC 7208 states it, 'independent domains.' a[4] is you (A records).
In this case, it looks like this:
v=spf1 a:mydomain.com ~all
So for example, you've got a domain name on Namecheap, and you're hosting on a VPS.
You've pointed your A record for @ to your VPS' IP address. That's all you need there, and all you need for SPF is a TXT record pointing for the current domain, @, and the string above, with the a mechanism pointing to your domain name.
SPF-compliant software will look up your TXT record containing the SPF string, see the domain, lookup its IP address, and match it to the server where your email is coming from, then allow it in transmission.
Popular mailers should tell you all this these days, but they don't, and it really sucks.
[1]: https://datatracker.ietf.org/doc/html/rfc1035#section-5.1
[2]: https://datatracker.ietf.org/doc/html/rfc7208#section-3
[3]: https://datatracker.ietf.org/doc/html/rfc7208#section-5.2
[4]: https://datatracker.ietf.org/doc/html/rfc7208#section-5.3
Work... technically... yes. If the definition of 'works' is: "I press send and sometimes my email reaches it's destination".
The spirit of the article IMO implies a definition akin to "I press send and my email reaches it's destination. And... my emails have some protection against spoofing attacks".
So in the spirit of the article, I would recommend your 1-liner takes a small yet significant change:
v=spf1 include:example.com -all
Now... let's talk about biggest barrier to entry for n00bs (myself included) in the self-hosted email game: non-blacklisted ip-address space. And blacklist here refers not just to public lists (SPAMHAUS etc) but private ones too (e.g., outlook).
Your configuration is trivially spoofable by setting Envelope Sender to something attacker controls (and thus it will pass SPF), while still placing your domain in the From header.
That said DKIM is not enough because that standard does not have a way to signal recipient that your domain has DKIM set up in the first place (and you promise that all your mails should always be signed).
It’s kind of funny but essentially hacker does not need to spoof DKIM because they can just omit it and recipient won’t be able to know that it should have been present in the first place. Btw, there was proposal to add a feature that could be used for this signaling but it didn’t get adopted, so DMARC is the only practical solution right now.
Wow, thanks. That definitely explains this mess.
Do you know why it didn't get adopted? I'd have thought that in a sane world all you'd need is to literally have a DNS record that specifies the DKIM info... pretty simple. Why on earth did people find it more convenient to do it in such a convoluted fashion instead? Especially now that there's ARC too, which I'd have thought shouldn't be necessary either.
If you read DKIM RFC it actually only specifies how to sign and verify the email signature, but does not mandate exact checks recipient should do to figure out authenticity. E.g. technically Microsoft could send mail from @gmail.com address and use "d=outlook.com" in the signature - the signature would be valid, but obviously it wouldn't make it authentic (meaning such mail wouldn't be authorized by Google). Although common sense dictates that there should be some sort of connection between domain seen in From header and the one that's DKIM-signing the mail (and most real-world implementations will do some kind of checks), RFC deliberately does not standardize these steps.
The intended standard that should have clarified these issues was ADSP, but it wasn't well received and now we have DMARC which handles both SPF and DKIM together (this is better for deliverability as mails that fail one of these checks might still pass another).
I somehow live without all those for what, since before they appeared. 2001 in fact. With my own MX.
They are just ineffective bandaids.
But in last 5 years levels of spam never even remotely approached levels what I remember from 2000s, when my job was actually fighting it (at an ISP).
All this tech (SPF, DKIM, DMARC) came and went ... somewhere, and there was exactly zero impact in spam rates on the handful of domains I now maintain, when I did take some time to implement them.
Nowadays I have a couple of my own domains, with nothing antispam-configured, and an gmail account, and you know what -- spam amounts almost exactly match.
Hence the conclusion - it's not worth it to invest time into it.
About everyone I feel a need to connect with understands that, so they don't use gmail, at least exclusively.
If a company trying to hire me can't receive email from me "because google" - then to hell with them, period. They are incompetent.
Although I must say there were no lost, rejected or diverted to spam messages from my deliberately ignorant MX to gmail for the last year.