Understanding SPF, DKIM, and DMARC: A Simple Guide | Hacker News Reader

Understanding SPF, DKIM, and DMARC: A Simple Guide

github.com

443 pointsmiles119 comments

I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated and doesn't include all the email services they actually use. (What this really tells me is how many small businesses are out there running with no in-house IT expertise or support of any kind.)

I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record.

It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS).

[1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...

Neat! I must try this.

What would you say the normal reception you receive from this email template is?

I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.

Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions.

Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.

>Well if you read the template

I read the template, that's how I spotted the acronyms that weren't spelled out. Like DNS on the second line, before you recommend forwarding.

>Overall I'm pleased with how well this approach works.

Interesting, I definitely would have thought it'd be ignored more often than not, but I might have to look into rolling out something similar. Thanks for the idea.

I think you're being a bit unfair here.

> If you do not have access to your company’s DNS records, please forward this email to someone in an IT role.

If you don't even know what DNS records are, I'd imagine you'd assume you don't have access to them and so forward them to the IT person as suggested. But sure, maybe he could also add ", or don't know what they are" to this line.

I'm not trying to be unfair or critical or anything. My first question was genuine. I intended my note about acronyms to be just that: a side note. The response I got was "If you read the template [...]", which it should have been pretty obvious I did. Then I got an explanation of the template as if I hadn't read it, which was a bit patronizing.

I think it's a good idea (and said so twice!). I was curious what the reception was like.

Sorry if my comment about acronyms was too much. It is a pet peeve of mine to see acronyms not spelled out, especially technical ones in a document intended for non-technical people. I didn't intend it to derail the conversation. Obviously it was taken in a way more critical way than I had intended -- my fault.

Sorry I didn’t pick up that you had read the template. I was just trying to give context for my answer without assuming or requiring anyone who might read it to have scrolled through all the code.

> ignored in most cases

I see no problem here.

That is really awesome. It can be easy to miss setting up SPF on every new tool.

Cool stuff! BTW, what's up with the license?

Including the whole license in the gist was probably overkill, but I particularly like Blue Oak for permissive licensing and would like to see it more widely used.

https://writing.kemitchell.com/2019/03/09/Deprecation-Notice...

Hey, English is not my first language so I'm surely missing something here but:

> Copyright

> Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it.

This sounds like the license specifically allows you to infringe on the contributor's copyright.

It's a permissive license. The licensor is saying “I own copyright in this work, which gives me the right to give you permission to do whatever you want with it.”

> If anyone notifies you in writing that you have not complied with [Notices](#notices), you can keep your license by taking all practical steps to comply within 30 days after the notice. If you do not do so, your license ends immediately.

I'm not sure I like it. Like, what if the notification of notice was incorrect? You lose your license anyway?

By default under most open source licenses, particularly permissive ones, violating the terms even by accident ends your license instantly, with no notice whatsoever.

Including a fair, common sense path to forgiveness severely limits legal risk for users, and is one of the things I like about the Blue Oak license.

Is it osi approved?

Does the script handle macros in SPF?

I've had a couple of other-company-IT-admins tell me that my MX is jacked because I use hosted SPF via proofpoint, and when they look up my SPF it looks like this:

"v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all"

A surprising number of mail admins don't understand SPF macros.

If I’m reaching for the script, it's because I’m already in a scenario where Proofpoint has quarantined legitimate email for failing SPF checks (we use Proofpoint too). So the script itself doesn’t do any analysis of the existing SPF record. It just shows them the existing record and tells them how to fix it based on the sender's IP for the email in question.

In defense of those who haven’t read the RFCs personally — I can count on one hand the number of times I’ve seen SPF macros in the wild, which holds true if I included yours.

Interestingly all Proofpoint customers too.

I’ve seen it more common to isolate services to subdomains and specify subdomain SPF records rather than use macros. This is my preferred approach.

I’m not hating on the macros. They’re just seemingly very rarely used. I know they’re on the table but I haven’t found a compelling use case in my own deployments.

I was on the receiving end of an automated version of this. However, when I looked into it, it seemed the problem (SPF record required more than 10 DNS lookups) was fairly common. I don't seem to have any other deliverability problems and my email and DNS is managed by some big hosting company so I assume it's not a real problem and didn't fix it.

It’s a real problem. The solution is just to isolate each service on its own subdomain. The only thing that should be listed in your top level SPF record is the corporate email for the domain.

The trouble is, it's full of stuff I don't understand because I'm not managing it, such as: include:_spf.qualtrics.com include:_spf.salesforce.com include:sparkpostmail.com include:spf.mailjet.com include:spf.protection.outlook.com include:_spf.myorderbox.com include:eig.spf .a.cloudfilter.net include:spf.websitewelcome.com

Yep, each one of those should be isolated to their own subdomain rather than being in the top level SPF record. Should have 8 total SPF records including the top level.

Easiest thing to do is to start by moving just 1 include to its own subdomain.

Awesome, mind if I send you an email solely to test if it gets through or if I get to be the recipient of your awesome script? ;-)

Sure, I think?

> At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated

And at the same time, I regularly get Spam/Phishing with perfect SPF, DKIM, DMARC, etc. The domains and IPs they use might get blocked within a day, but of course, these people have no problem getting others.

And although I have set up my MTA perfectly, my mail gets refused by MS/t-online/etc., because I don't have enough "sender reputation". In e-mail, we have an oligopoly of a few big mail providers, and in the end, they decide which mail gets delivered and which isn't, and to me it looks like they give a rat's ass about SPF and DKIM, and probably rightfully so, because most spammers are probably better at configuring MTAs than your average mail admin.

t-online uses a global whitelist, which is pretty stupid for e-mail. sometimes it helps contacting them, other times they refuse to resolve it for arbitrary reasons (not because of actual spamming)

t-online told me I needed an imprint on the website that's reachable under my domain. Seems to be some misunderstanding of German law (German commercial websites need an imprint, legally, but t-online also apply this requirement to private domains).

Last month they unblocked me even though the website is blank.

It is more complicated than that. There are more criteria for when you need an imprint:

(1) any kind of journalistic content on your site

(2) any kind of financial gain from showing ads or making ads

(3) organizing any kind of group of people active on German territory

(4) running a business website

There might be more, but those are the ones I remember from reading the paragraphs a while ago.

And these are, of course, vague, which means that even something like "my favorite restaurants in Berlin" could be considered an ad, or any kind of comment on politics might be considered a form of journalism.

I dislike these rules, because they basically kill German blogging scene. Not so many people want to run a blog and have every idiot on the Internet know their personal address. And few bloggers want to rent a digital office or actual office, that will send mail to them (an indirection). The German law in this respect is terrible and working against a free Internet and against freedom of voicing your opinion. It works greatly in favor for tech giants, because people resort to putting their blogging on Facebook, Instagram and other disservices. It is very anti-decentralization.

> have every idiot on the Internet know their personal address

It doesn't have to be your personal address though.

I'm somewhat mixed about the details of the law, but requiring businesses to make it clear who you are dealing with makes sense to me.

Of course, if you run a business. But then you might even have an actual physical office address for work related stuff and not have a problem giving that to random strangers on the Internet. If you are a private person simply wanting to run a blog and talk about whatever you like, then it sucks. It does not have to be your home address you are sharing, but the other options will cost you money. And no, a postbox will not be sufficient, as decided in previous court cases. It must be a "ladungsfaehige" address. There are some businesses selling services to have a virtual office, with an actual address, which then send your mail further to your actual address and notify you via e-mail and all that. Whether those are really an acceptable option when it comes to the law is a bit unclear.

On some level I can see it being a benefit to the big providers to only accept email from other big providers, as it would incentivise people to buy email services from them, because only email sent from the big providers would "work".

T-Online has a simple whitelist approach, and it is usually enough to just drop them an email. I did that back in 2014, asking for my private mail server to be added to that whitelist, and I received a positive answer within a few hours.

Maybe that was possible 10 years go, they now require that you put up a web page for your domain with a valid German imprint (most importantly: your full contact information).

To be fair, I had that in place 10 years ago (and if you already have a mailserver, it's trivial (as in: MUCH easier than to set up a mail server) to host a small imprinted HTML page).

> most importantly: your full contact information

Note that unless T-Online has additional requirements here this doesn't need to be your home address but only a valid mail address through which you can be reached.

In Germany, you might be required by law to publish your address for everyone to see.

Original German text of the law: https://www.gesetze-im-internet.de/ddg/__5.html

Google translation to English: https://www-gesetze--im--internet-de.translate.goog/ddg/__5....

I work at an IT provider - we see this daily. Have to whitelist to keep the customer happy, usually the other end is a 1 or 2 person business with an old hosted Yahoo system or similar

> happy

or misguided?

(I know the pain.)

If we care about keeping open protocols like HTTP and SMTP alive, we need to overhaul DNS.

Or at least create a simplified common abstraction layer.

It’s the most inherently user-hostile thing I’ve ever encountered - and I’m only just now starting to understand it, even though I’m almost 20 years into dealing with it.

My favourite are queries "why are you rejecting my email?"

Ehh... Because your dmarc policy told us to?

How do I make use of the gist?

I do

    (load "spf-fail.rkt")

But afterwards none of the definitions are available in the Racket REPL. Maybe I need to (require ...) something? I do not see a module definition, that I would need to import and

    (require spf-fail)

fails.

If you’ve just run "racket" from the command line to get a REPL, you would use

    (require "spf-fail.rkt")

to import all the bindings from this file (assuming it's in the same folder you ran racket from). The “Module Basics” chapter of the Racket is a good quick explainer of how this stuff works in Racket: https://docs.racket-lang.org/guide/module-basics.html

Another method would be to open the file in DrRacket (or VSCode or Emacs or whatever editor you have set up with a Racket plugin/lang server) and just "run" it in the REPL.

There are comments at the top that explain how to use it once it's loaded:

    ;; Generate a form email to let someone know their SPF records are misconfigured for their current email provider.
    ;;
    ;; Run (fill-report "domain.com" "1.2.3.4") where the 2nd arg is the sending email server's IP address.
    ;; It will copy the completed report to the clipboard for you.
    ;;
    ;; Only works on Windows for now.

Personally, I have this file incorporated into a larger package (not published anywhere) for producing canned responses. With that package installed I can do this at a command prompt:

    raco canned spf domain.com 1.2.3.4

Yeah, the only thing you need for simple SMTP services is:

- with DNS, the simplest as possible SPF record.

- Without DNS, (aka pure IPv[46] SMTP), your have implicit SPF: instead of querying the DNS and parsing the SPF record, you parse the mail header to check "reply-to/from/etc" fields (the appropriate fields) for the sending SMTP IPv[46] address, that to perform spam scoring.

What I really want to see is a guide for SPF/DKIM/DMARC oriented towards people writing apps that send email using other people's domains. I have dealt with so many ticketing systems and marketing platforms that do not understand the roles of SPF/DKIM/DMARC at all.

Things like, insisting we need to include their SPF record in ours, even going so far as to scan the SPF record for the include, only to find out they use their own domain in the envelope address (which is what I wanted them to do in the first place).

Or not distinguishing at all between envelope and header addresses and using our domain in both. Which of course means they're not tracking delayed bounces.

It really becomes an issue with larger orgs where everybody wants to use the main domain for brand purposes and subdomains are just totally frowned upon for whatever reason. If you just leave my SPF alone and rely on DKIM, it means you can still pass DMARC and track bounces properly. Hell I'd be fine with making subdomains for the envelope address that lists your infrastructure in the MX records but again, eyes really start to glaze over when you say "envelope address."

Basically what I really want is a guide that boils down to: if you're not their primary email provider, then don't touch your client's SPF record.

I recently set up a mailchimp tenant for someone and was surprised that their email authentication wizard pretty much began and ended with DKIM. I'm way too used to b2b solutions that want the equivalent of a chmod+999 before it can run a hello world.

In my experience this approach is becoming more common, but slowly.

I still configure my default apex DMARC records (in most scenarios) to enforce strict alignment on both SPF and DKIM, but I’ve been relaxing that on a case by case basis or overriding the apex DMARC policy at a subdomain level and only using DKIM where supported.

There seems to be incorrect information out there about strict alignment requesting more strict checking. Strict vs relaxed alignment is just about the domain being able to send mail from subdomains without extra DNS records on the subdomains (relaxed) or not (strict). The envelope from (for SPF) and d= domain (for DKIM) must match the DNS records used. If you don't use SPF for DMARC you don't use SPF for DMARC and it doesn't matter if the DMARC SPF alignment is strict or relaxed (there is no way in DMARC to require both DKIM and SPF to pass). It is still a good idea to always pass plain (non-aligned) SPF (just based on envelope from) since sometimes this is checked independently from DMARC.

I am slightly confused, and perhaps misunderstanding your framing.

I do use DMARC and SPF in the fashion you described. In my environments I typically need to take every measure to ensure only authorized services/servers/senders are sending, via authorized hosts and IPs, and this often changes based on subdomain, so that’s why I use strict alignment. Personally I strive to keep various services strictly separated by (sub)domain.

I thought it might just be ambiguous language. It wasn't clear to me what you were trying to say and it could be read as implying what I have seen (incorrectly) stated in some DMARC descriptions elsewhere. Just trying to prevent any misunderstandings from anyone reading the thread and improve my own understading by checking that I got it right before posting. Thanks for clarifying what you ment :).

On a slightly related note, Michael W. Lucas[1] is working on an upcoming book entitled "Run Your Own Mail Server", that will be published shortly (there's a Kickstarter campaign as well[2]).

I attended his tutorial and talk at BSDCan[3] this year and both were excellent. I highly recommend buying the book when it comes out (or supporting the Kickstarter), it will go through all the gory details of setting up and running a mail server, and best practices, including a ton of material on SPF/DKIM/DMARC.

(P.S. I have no affiliation with the author or the book in any way.)

[1] - https://mwl.io/

[2] - https://www.kickstarter.com/projects/mwlucas/run-your-own-ma...

[3] - https://www.bsdcan.org/2024/

Looking forward to this. First thing I ever ponied up on KS for.

I don't even run a mailserver, I'm just hoping it will take a bunch of the guides that have been floating about on the web, consolidate the sharp edges, and make sure its up to date.

I also hope it has some discussion on troubleshooting. Like dealing with blacklists and what not, folks always talk about that, but I've never see it documented what is actually done to resolve these problems (Like who do you send an email to, how do you even find out who to send an email to, etc.)

Disclaimer: My startup is made to get user domains automatically onboarded to applications, DNS-related things such as SPF/DKIM/DMARC. It's a "Stripe-for-DNS" called cloudvalid.com.

I like the quality of this SPF/DKIM/DMARC guide. This is the industry I started out in, and I actually wrote the guides for SendGrid, Amazon SES, and a few other email products. I don't mind saying that author has done a better job at this than me.

That being said, I see SPF/DKIM/DMARC guides like this pop up with some regularity, but users continue to have the same level of comprehension as before. I think the nature of this problem is not one that lends itself to being solved by guides. It's the sort of problem that a user is really only faced with once, which means that they're not getting the repetitions in to warrant any long-term comprehension.

I'm naturally biased here, but if you're onboarding users with SPF/DKIM/DMARC to your application, it's good if you can just get them setup with automation.

> My startup is made to get user domains automatically onboarded to applications, DNS-related things such as SPF/DKIM/DMARC. It's a "Stripe-for-DNS" called cloudvalid.com.

Interesting. Not lot of startups out there talking about Domain Connect on their front page. Seems like a decent amount of overlap with my project TakingNames.io[0]. Feel free to reach out if you want to talk shop.

[0]: https://takingnames.io/blog/introducing-takingnames-io

Just today, someone sent me the link to a great tool to debug dmark issues: https://www.learndmarc.com/

Just tried it - you're right that is really good. Thanks.

Fantastic!

Is it just me who finds this tool horrendous? Instead of just giving a report, it makes words fly on the page... Certainly it looks cool and perhaps appealing to some younger generation, but if it is tech analysis, you would expect clear and concise report, not some flying things with explanation like I am 4

It's called learndmarc, not testdmarc. As someone who wasn't familiar with the technologies involved, I like that I actually learned something while using it to debug my email issue.

uh, you are right, super cool!

sysadmins will love it

https://www.mailhardener.com is another great "validator" and more for a personal email domain

This is a great guide but from my experience, even if you configure it 100% correctly, email services like Gmail may still classify your emails as spam for no apparent reason while not being on any IP or domain blacklist. I tried for hundreds of hours to get around it with no avail, and my emails to Gmail always went to spam unless it was a response to an email from a Gmail address. Had to go back to a 3rd party hosted service (iCloud) because of it.

I have had a Gmail account from the days when it was invitation only. The inbox contains spam and my test emails and nothing else!

I've run tiny smtp systems for 25 years or so. It can be done. I am based in the UK but at least one of my domains is a .net jobbie, so nominally American. That one still works fine and it is my (ltd) company domain, so all good. The MX records etc have moved around a bit but always very carefully.

It all starts around the IP address you are using. Is it "tainted"? is it in a tainted block? If it is then you need to either go elsewhere or clean it up and that takes a bit of time. By clean it up I mean apply for removal from the usual suspect's blocklists - Spamcop (lol), Spamhaus and all the rest that you can find.

Now setup PTR records. That has to be done by your ISP. If they can't do it for you, then find a new ISP. If you can't get PTR records to match A records then you may have to give up. One of the first checks an anti spam system will do is reverse look up an incoming IP address and compare it. Also that should match the HELO/EHLO announced by the SMTP MTA:

SMTP connection from IP address 12.13.14.15 HELO (my name is) smtp.example.co.uk

Receiver will check: smtp.example.co.uk == 12.13.14.15 AND 15.14.13.12.in-addr.arpa == smtp.example.co.uk.

Everyone gets their knickers in a twist about SPF, DKIM and DMARC but if you do not get the prior basics of IP -> A -> HELO -> PTR sorted out first then you will fail sooner or later. I also recommend that you ensure your MX records (receiving) match up too with your sending records. It means you can use mx is SPF, for example.

If you have multiple internet connections and IPs then be absolutely certain that your inbound and outbound IPs for SMTP match up.

Sorted all that? Cool, now proceed to SPF.

Most people fail at the PTR stage. If your ISP will not do PTR for you then you are probably screwed for self hosted SMTP. If you cannot change ISP to one that will, then you are really screwed. Sorry. In that case you will have to engage a service that will route SMTP on your behalf. It won't cost much but you won't own it and you will have to pay someone to do it. Soz.

Reverse pointer is pretty easy with some hosting (Linode) and painful with others, but that's pretty basic knowledge. Same with managing IP reputation. Heck, mail gun helps warm up IPs for you (but if you're not email marketing it's ridiculous to maintain that).

What really gobbles my bobble is BIMI. Even without the paid-for certificate ($1500 is absurd), you can set it up to show your logo, and works on some providers (like yahoo). But careful, you have BIMI without the cert set up? Gmail spam-cans it.

Same with pgp, if you include your signature a lot of providers will immediately increase it's spam rating, usually high enough to land in spam (+7 pts usually), even though I doubt any spammer or scammer is inviting you to encrypted chats.

Email is broken because we all signed up for Gmail and didn't know better at the time.

It's getting pretty expensive to rent one IPv4 address per domain these days. You also don't always control every address in a block, which means there may be nothing you can do about your reputation no matter where you go.

You only need one IP per MTA not per domain. I have a "vanity" email system that I run at home. I run it for my mates too. I have around 10 domains inbound. It all works fine.

SMTP and SIP are often held aloft as fucked up. My Dad's home telephony runs off a RPi and a Yealink DECT station and a dynamic DNS.

The modern internet might look a bit fucked up if you only look at the X/Facebook/webby wankery stuff but the real internet is functioning quite happily.

>Spamcop (lol)

What is lol about Spamcop?

Came here to say this, plus add a little personal insight to the future of email.

I've run 5 or 6 different mail servers over the past 10 years. Originally before O365 I was an exchange admin, then postfix, iRed, mailcow, mail gun, you name it. Hosted on every cloud provider, even in our colo with part of a private /24 allocation with good reputation (built since 1997, gawdamn). Every sort of header combination, tls setup, and no blacklists. Always 100% alignment, including strict rejection policy (best results even over quarantine).

Does not matter, if you're sending from custom domain not handled through a big name, expect the spam box with Gmail. Yahoo and Outlook are fine, but Gmail is the bane.

I've spent maybe 100 hours of my own over this last year and know what I realized? Nobody cares about email anymore, except for automated account management stuff (login, PW reset). Businesses pay the $3 /mo / seat for fastmail and don't think twice.

But the current trend is toward social chat (discord or Whatsapp) and most the people who own an iPhone just use their apple ID email for everything.

Although I am a fervent supporter of open protocols and believe email (with pgp signing) is an awesome long form communication format... Face it, it's going the way of the fax machine.

Perhaps it is a volume issue? You need steady and significant volume of emails to maintain reputation at gmail and friends.

Guilty until proven innocent, an excellent initial position.

I’ve had to relax my SPF record to include the entire mail pool of my ISP to be able to send to anything hosted by Microsoft. I tried to liaise with them directly, and through Linode, but they refused to exclude the IP from their opaque blocklist. Their proposed solution was to change the IP of the VPS, but that’s just agreeing to play whack-a-mole with a bad faith actor.

There should be a path to greater transparency and accountability from the SMTP cartels, but I’m at a loss as to how that can manifest.

Did you send out mails directly from your ISP? (Than I could unterstand Microsoft.) Only your MX must send out mails. And only it should be in the SPF record.

It would be crazy of Microsoft to look at all Received-Hearders and want everything mention there in the SPF record. If that is what really happens, than you should exclude this information from the header. (mask-src on Opensmtpd. Pretty sure Postfix has that option to, but haven't used it in a decade, so I can't tell you the syntax.)

I originally had only my MX hosts listed in my SPF record, and configured to accept them exclusively. Microsoft unilaterally blocked the entire /23 my VPS resides within, and refused to exclude my configured IPs from their block.

The only way I’m currently able to deliver anything to domains hosted by Microsoft is to expand my SPF record to include my ISP’s mail hosts, and route delivery through them.

Microsoft, and the other SMTP cartel thugs, are undermining the protections these protocols were designed to provide.

Email as a communication method with your friends/family — absolutely, this has been dead for over a decade I’d say.

However, email has basically evolved into the way you communicate with “systems” and I’m kind of happy about it. Communication with companies outside your network, e-commerce accounts/purchases, communication with government systems, schools, banking, airlines, concerts/events, restaurants, etc. Hell, even RSS is now basically in email — newsletters are growing fast as a medium, not shrinking.

You just book a hotel in Nairobi? It’ll be in your email. No other communication method even comes close for this use case.

Social/chat apps will never unseat this because they’re social. Like nightclubs, the trendy ones come and go. Come back when you’ve set up an interoperable network of virtually every person on earth. Then we’ll talk about email being dead.

> Email as a communication method with your friends/family — absolutely, this has been dead for over a decade I’d say.

Email is the primary way that I stay in touch with my extended family and friends. At least for us, it's very much alive.

We need more email diversity. Use your own email servers as often as you can. Monocultures of Apple/Google/MS is deterioration for the Internet.

And you don't need to run your own server in order to escape the big players. I have been a happy Fastmail customer for many years now.

I second Fastmail wholeheartedly. I moved multiple self-hosted (Postfix and Dovecot) environments to Fastmail and haven’t looked back. The only thing I lost was the ability to manually configure an address to bounce back.

Sending and receiving as a wildcard alias is fantastic, and Fastmail allowing that was the reason I finally moved. I held out on that feature for a long time. (You can also do this in Exchange Online if you want to run a convoluted and officially unsupported configuration.)

I only twice had any kind of RBL issues, and one was a motivating factor in moving. I also got tired of worrying about patching and CVEs. I do that enough for my work.

One of the reasons I decided to self-host emails for my domain names. There's no reason it should be hard to host your own email in 2024 even if you're just mildly technical.

An Ansible playbook that sets up Debian to host email, all the dependencies from the default Debian repositories.

https://github.com/programmer-ke/replatform/

I would add https://www.mail-tester.com/ to the testing tool list

the free version of https://eu.dmarcian.com (which is great btw, but costs a bit out of my budget) is good, too

this is good too https://www.20i.com/blog/dkim-demystified/

dmarcian won't load in NL:

> On February 20th 2024, the Hague Appellate court instructed dmarcian to block dmarcian.com visitors who originate from Dutch Internet Protocol addresses.

Apparently they had some trademark dispute with their local representative. The story on their website seems to be missing some key details to understand the whole picture.

oh my

that's odd

thanks for telling us

> If you are invovled in developing, supporting, or maintaining an application that sends emails, this guide is a must read.

I would also say a guide like this is helpful whenever you are using a custom domain with an email service, and therefore need to set these records yourself. Okay, you might not need to have an in-depth knowledge of these concepts, but it's certainly helpful to understand why your email provider is telling you to set all these weird DNS records.

Many of our emails were blocked because our SPF record was correct for our domain, but not correct for the Return-Path domain. There is no mention of Return-Path in the guide, but it's essential to get this right.

https://easydmarc.com/blog/how-does-an-spf-record-work/

> For example, a business might send promotional emails to its customers. By implementing SPF, DKIM, and DMARC, the business ensures that its emails are not marked as spam

But such emails can very well be (and in my experience usually are) spam. My spam filter will take the lack of implementation of these as a strong signal that the email may be spam, but does not think that just because these are implemented, the email isn't spam. As it should.

Just wanted to call this out because the article strongly implies that implementing these will let your email bypass spam filters.

I run my own email server, and I'll add that SFP /DKIM and spam filtering should be done at the connection level during SMTP session. Your MTA should not accept a message for delivery that has been filtered. This ensures that the sender of false positive gets a notification from his MTA and also, in my experience, it works very well to discourage spam senders from trying again as they think your MTA is broken.

I am guessing that Apple is one of the top 10 email providers.

Why don't they send DMARC reports?

Every little podunk email provider seems to be able to send me DMARC reports, as well as Microsoft, Yahoo!, Gmail, Xfinity, and AOL. Every morning I get a pile of them. But never a peep from Apple.

They like to reject our email for inscrutable reasons. We don't send bulk email. We only send email (almost entirely transactional) to about 5,000 paying customers who have accounts on our website.

Two days ago I was however delighted to receive a reply to an email I had sent to them on May 1 asking for assistance and clarification on why they block us. They told me that the website of one of my wife's employers (not our website or domain), featured in her email signature, was hosting malicious software. It isn't on any blocklists and I can't find any evidence of this currently being the case. Maybe it did at one point and they cleaned it up? I suspect Apple blocks and never revisits if the reason is still valid.

When I first implemented the above the next step was going to be ARC.

Does anyone have thoughts on ARC?

https://www.validity.com/blog/how-to-explain-authenticated-r...

It's only really needed if you need to robo-forward stuff between domains. For example if you set up a domain but want to receive emails to your usual gmail.

I noticed that cloudflare's email forwarder uses an ARC record and it works a treat.

I recently noticed this when debugging email delivery issues for a family member who have their own TLD, but forward everything to Gmail.

Unfortunately, the mail server of the forwarding domain doesn't seem to support ARC, so Gmail frequently throws away everything that doesn't have a DMARC header, since without DMARC the only other option is SPF, which doesn't work for forwarding.

A lot of small businesses in my area have been bit by that this year. I have to give hostgator/bluehost/godaddy etc kudos for having an email forwarder work reliably for so long but I wish they were more proactive about getting their customers compliant with this.

Also it's kinda messed up how much of the small business sector is relying on AOL webmail to operate.

I'm thinking I will add it as part of next upgrade. My domain has so few users and we are chill enough to work through email delivery issues. My understanding at the time was that the receiver verifies that the sender sent it.

My current delivery settings are strict so that only my server can delivery our email. I would think I could implement ARC in less strict manner and tighten it up as it becomes more common.

Does that seem reasonable? Any better ideas.

I also wrote a similar post a while ago, after someone was sending emails using my domain name: https://www.uxwizz.com/blog/stop-others-use-your-domain-emai...

Tangential, but what is the contemporary go-to for standing up a mail server these days? The last time I had to do so was a decade ago.

I remember Mail-in-a-Box being popular at one point, wondering if that's still the case.

Really depends on what you want to get out of your server. Most easy to use server software I know seems to be geared towards personal use and small organisations.

Mail-in-a-box still works, though last time I checked they were on quite an old Ubuntu LTS release. There are a few pre-packaged docker containers too (i.e. docker-mailserver) which seem to be popular. I myself use Mailcow, but that's pretty heavy for "just" a mail server.

There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go?

> There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go?

maddy or mox?

For a business? You don't. For hobby/fun? I still wouldn't. I tried years ago and you're just going to fight blacklisting constantly. If you're on a residential/consumer internet service, trying to set it up in your home lab, forget it.

I’m partial to https://gitlab.com/simple-nixos-mailserver/nixos-mailserver

Does require some NixOS knowledge, but is nice for upgrades.

There's a bunch of options: https://github.com/awesome-selfhosted/awesome-selfhosted?tab...

I personally use docker-mailserver https://docker-mailserver.github.io/docker-mailserver/latest... because it's a pretty traditional stack (postfix+dovecot+sieve etc) just already containerized and configured, so there's a lot of info already out there on how it works

The number 0 requirement that you have to solve regardless of stack though is to get an IP with a good reputation. I've got Comcast Business to my apartment, which I think is probably the best way to get a good IP since it's relatively difficult for spammers to have used it in the past. Alternatively, relay everything through Mailgun/SES/whatever

Recommend Stalwart

https://github.com/stalwartlabs/mail-server

(No affiliation to the devs)

My problem with SPF (& co) is redirections.

I have email redirected from other domains into my (Gmail) inbox. For it to arrive, I use SRS, so the email is properly aligned and always makes it into my inbox. The problem is that some of that email is malicious. I have a choice of dropping those mails, and I never see a trace of it in my inbox, or forwarding them with SRS, and they look to Gmail like 100% perfectly good mails sent from my own domain (but still potentially malicious). It's annoying.

> It's annoying.

This seems very much like a problem you have created for yourself.

I know, how dare I use other domains than gmail...?

Why post a comment at all, if you have no insight. It's useless and insulting.

Why post on a public forum if you dont want, or even get insulted by, other peoples opinions?

Its nothing to do with using domains other than gmail, its that you said you are actively relaying all mail into your gmail account and rewriting the sender as yourself, but then its annoying that spam gets marked as valid mail from your domain. Thats nobodies problem other than your own setup, and there are loads of other ways you could do it. But the way you chose to set it up is 'annoying'.

Sorry to insult you, but I feel that warrants letting you know you caused your own problem!

If your opinion is "you suck" it's called an insult.

You could have ignored my question, you could have explained why I'm doing it wrong, hell you could have posted a random setup guide from Google and downvoted me. Instead you post this attack with no content, justification, or insight. You are being an ass. You are allowed to be, but I'm allowed to call it out.

And if you think I set my email up in a sub-optimal way, just so that I could ask for advice on HN, with the goal of pissing you off... you are delusional.

Since you seem so well versed on how the best way to not be insulting, maybe you should follow your own advice and stop insulting other people. Just a thought, but maybe if you tone down your responses and be a bit less insulting yourself then maybe karma might do you a favour.

Take a look back at the thread. This is what I posted:

> This seems very much like a problem you have created for yourself.

Thats it. Nobody said you suck, nobody insulted anything. Just pointed out that a problem you said was annoying was caused by your own doing. If you think that is an attack, and warrants calling me delusional and an ass, then you need to look very hard at your own reactions to things. You resorted to namecalling and mudslinging just a bit too quickly.

If you do SRS correctly it will not pass DMARC alignment for your domain but it will pass plain SPF which does not have the DMARC alignment check and is sometimes checked independently from DMARC. If the sender included valid DKIM it should pass DMARC for the sender's domain as long as you don't alter the signed parts of the message (unless possibly if they do something annoying like sign the absense of X-Forwarded-To). Google also wants you to use ARC, add X-Forwarded-{To, For} headers, avoid forwarding spam, and use a different IP address or domain for forwarding vs sending mail from your domain. Some email providers let you indicate that you trust particular ARC forwarders but I don't think Google uses it that way.

https://support.google.com/mail/answer/175365?hl=en

I don't know why Google want to force forwarders to do spam filtering.

You're probably right about the terminology, sorry. My problem is that a lot of legitimate senders have failing (or soft-failing) sender setups, so I can't have my forwarder just drop all that (I'm not even sure my registrar-provided forwarder has that option).

Another option would be to have another inbox on the domain and have Gmail fetch with POP/IMAP, but many domain registrars don't have that service. Or is that what most people do?

For receiving custom domain email long term I think the usual thing would be to have the chosen email provider manage all email for the domain (setting up MX records) and not forward long term. That would require a paid account with Google it looks like, but at the point you are paying for your domain that seems reasonable (I'm not sure how much they charge). The fetch options are usually either time limited or expensive from what I've seen. I wouldn't be surprised if registrars don't forward correctly according to Google's instructions, they just want the least expensive thing that lets them advertise email as a feature, but maybe worth complaining anyway if they don't.

For personal accounts I don't think you can convince Google to accept mail from incorrectly configured senders, you would need to use a different service if you must receive it. For business accounts they are less strict, although Google still randomly drops mail and I wouldn't personally use them for that reason and others (I think they have good suggestions for how mail should be set up, though). I'm not trying to sell you anything and haven't even tried them yet (and have some questions I haven't asked yet) but purelymail.com is one that seems promising to me in general and is inexpensive with no extra fee for custom domains (if you aren't attached to the way Google handles mail; I'm guessing the archive search and general interface will be more clunky). I'm guessing you can configure SpamAssasin to not strictly reject mail from incorrectly configured senders as long as they aren't on spam lists (they say they reject those immediately), but I'm not sure.

Thinking about it more I'm guessing Google requires forwarders to check spam due to not wanting customers to configure trusted ARC forwarders like Microsoft does and so not forwarding spam makes it easier to convince them the forwarder isn't a spammer trying to blame others for the spam (they might also compare forwarded spam with what they receive directly). If a forwarder uses ARC and can convince them they aren't spamming then Google should trust their SPF assessment if you just mean senders who only have SPF configured. But if you want to receive email from old timers who insist that email should have no way to detect spam there may be no way to do that with Google (or a number of other providers).

FWIW: Postmark (a service I like using) offers a free DMARC monitoring service.

https://dmarc.postmarkapp.com

I needed to set up an email for my own hobby domain and spun up a little Ubuntu server and ran this:

https://mailinabox.email/

Worked a treat and did everything. I was sending and receiving email within a few hours.

These kinds of articles pop up on HN all the time...

Give me a mail server that can use LE for certificates and I'll gladly give DKIM and DMARC a try...

Sendmail

I can't believe sendmail still exists... worst configuration format ever.

You don't need public certificates for DKIM, it uses privately generated ones for as long as you want to keep them. (A security researcher recently found quite a few domains using weak DKIM keys generated by buggy Debian OpenSSL, more than fifteen years ago.)

I could swear at one point you needed one, but I just half-setup opendkim and it generated one without me needing to make one by hand... when I get around to updating the DNS on my personal domains, I guess I'll see how things turn out.

is this good?

--- SPF --- RFC5321.MailFrom domain: example2.com Auth Result: PASS DMARC Alignment: PASS

--- DKIM --- Domain: example2.com Algorithm: rsa-sha256 Auth Result: PASS DMARC Alignment: PASS

--- DMARC --- RFC5322.From domain: example2.com Policy (p=): none SPF: PASS DKIM: PASS DMARC Result: PASS

The DMARC policy is none. Meaning if both SPF and DKIM fail then nothing should be done about the email. Now, many email security gateways and spam filters will just have rules automatically blocking anything that fails SPF regardless but you want to eventually get to `p=reject`.

Start with something like `p=quarantine; pct=25` to have 25% of reported DMARC policy failures be marked for quarantine, review the reports after a week and then ramp it up to 50%, 75%, 100% every few days. Then if your domain is not having a significant percentage marked for quarantine in your DMARC reports after a week or two, switch to `p=reject; pct=100%` and continue to monitor the reports to make sure everything is good.

DMARC is not bulletproof to people using your domain as spam though because even with a reject policy, if SPF fails but DKIM passes, DMARC will pass or vice versa. It helps curb abuse and takes 15 minutes of effort to set up once but still is not enough to kill spam.

Reply on news.ycombinator.com