DMARC Secured Email Identities but Broke Mailing Lists | Hacker News Reader

DMARC Secured Email Identities but Broke Mailing Lists

learntemail.sam.today

82 pointssamtoday50 comments

Forwarding email with outlook.com/Office365/Microsoft Exchange also breaks DMARC. Exchange Server sometimes modifies the headers of emails when forwarding them, invalidating the DKIM signature, so then a DMARC policy rejects the forwarded email. Apparently Microsoft have a fix in the pipeline for this, but it's been taking ages.

More info at: https://blogs.msdn.microsoft.com/tzink/2016/05/19/why-does-m...

The primary problem is that DMARC is fundamentally flawed, and was not enacted using a standards process that respected all of the stakeholders. As a result, it fundamentally becomes a matter of power politics.

If there are a bunch of people who need to participate in a particular mailing list --- say, IETF mailing list or the Linux Kernel development lists --- more than they need to stick with a particular mail provider, it becomes possible to say to them, "you want to participate in our community"? Change mail providers.

In the cases where a mailing list community badly needs the Yahoo users, Yahoo can dictate to the mailing list --- change your mailing list software and inflict pain all on your mailing list users, or you don't get access to our e-mail user community. (And rewriting the from field has all sorts of bad effects, including corrupting contact databases that auto populate based on names and addresses in the from field, making the mail summary index useless, breaking reply-to sender, etc. So there is real pain involved here.)

Part of the issue was that DMARC was originally intended for domains that only sent official announcements (e.g. for credit card companies or banks) and where employee e-mails came from a different domain. For that original use case, DMARC worked perfectly well. Apparently Yahoo then decided it had a terrible SPAM problem, and decided DMARC was a blunt instrument to use, and inflicted on consumer e-mails. Other companies didn't think ahead, and used the same domain for official e-mails as their employee e-mails, and decided that protecting their users was more important than inconveniencing their employees.

This is why it's ultimately all about power politics. If you want to participate in Linux Kernel development, you need an e-mail address that won't arbitrarily cause your e-mails to be dropped (for example, so Linus Torvalds doesn't get your pull requests). Despite this, growth in Linux Kernel Development continues to be growing, which means that when choosing between the inconvenience of changing or using an alternate e-mail provider, versus participating in Linux development, developers choose the former. But that only work because Linux has the economic power and clout to get away with it.

Heck, over ten years ago, refusing to buckle in to crap e-mail systems forced IBM to allow its Linux Technology Center folks install a standards-complaint IMAP server instead of using that abomination caused Lotus Notes which the rest of the company was forced to use. (And this was probably a better demonstration of the power of Linux, given how much IBM was stubbornly attached to Blotus Goats.) But make no mistake. This is Trump style, power politics. It doesn't hurt those with a lot of economic power, but if you're some tiny, podunk church mailing list, or some other group lacking in economic power, you're screwed.

DMARC: making email "great" again.

I stopped reading when it referred to SPF as "Sender Protection Framework." As usual, the discussion on HN is far better than R'ing TFA.

We recently had a collision between the practices of one of our vendors, the silly way that .edu's tend to forward mail, and Gmail's DMARC policy. The vendor is doing everything "right" in terms of their own (completely transactional) mailings on our behalf but some still want to blame them for poorly forwarded messages sent to .edu addresses that subsequently get swallowed up by Gmail and are never seen by the authors or referees.

I've seen the same issue with an edu address using office 365. The user set his mail for our org to forward to his .edu address. The .edu address was set to forward to his personal Gmail. The Gmail at the end of the chain was rejecting the mail because the edu didn't rewrite the from address.

Authenticated Received Chain (ARC) is being developed to work simultaneously with DMARC and help provide relief to mail agents that break DKIM by allowing ARC aware receivers to sign authentication results that allow downstream processors to review what the initial results were and make their decisions.

Details at http://arc-spec.org/

> For many years, there was no real way to verify that you really got the email the person that the From header states.

DMARC doesn't really assert your identity---it asserts the identity of the server.

Remember that you can also use PGP to assert _yourself_, and this works well with mailing lists (your mail client will hopefully distinguish between the signed message and the mailing list footer, which is unsigned). It also persists---if a mailing list is stripping DMARC headers, then that doesn't help you any.

PGP signatures can't be used to prevent delivery of scam/spam email.

Yes, you're right.

I modified my message to make clear what I was commenting on.

So what did LKML end up doing?

Edit: given the Ts'o response above I guess they enforce the sender to not have DMARC.

"There are people with google.com addresses that need to use non-Google addresses in order to participate on the Linux Kernel Mailing List."

I work on Gaggle Mail which is a group mailing list provider that avoids all this by only using a From address that we control. I do think stricter addressing policies will become the norm in years to come which is why we took this approach.

More details here: https://gaggle.email/how-emails-are-addressed-and-sent

As someone who thought it'd be a good idea to run their own mail server, I found that without SPF/DKIM/DMARC your mail gets identified as spam a lot. Having DMARC just means you get told about it.

That is nonsense. I run a mail server. I have SPF, but no DKIM/DMARC. I have no problems with mail deliverability.

In my experience there are a lot of urban legends around this topic.

The problem comes if the sender has a DMARC reject policy, and he or she sends e-mail to a mailing list which then reflects the mail to a recipient which honors DMARC, the recipient will bounce the e-mail message, and never see it. In some cases this will cause the recipient to be automatically unsubscribed from the mailing list.

If you're not using DMARC, and you're not running a mailing list, then yes, it won't affect you. But it's no urban legend.

I have had (and continue to have) perfectly innocuous emails land in the spam folders of others with gmail.

So have I - including when sending from another Gmail account. Their spam filters can be fickle.

Running without DKIM/DMARC exposes your domain to a suite of spoofing attacks that aren't mitigable by SPF alone. A spammer can send mail "from" your domain and over time this activity may lessen the reputation of your domain and/or IP/netblock—even if it's just Gmail users manually marking such messages as spam—making your subsequent outbound, legitimate messages more likely to be flagged. If nothing else, the backscatter gets annoying! DKIM, DMARC, and ADSP are worth setting up for these reasons, and the reports provided by e.g. Postmark[0] are invaluable for understanding how your domains are being abused.

0. https://dmarc.postmarkapp.com

Are you on IPv4 or IPv6? I've found that IPv6 is much more likely to have problems with deliverability than IPv4. When my own server was IPv4-only, I never had problems, even with only an SPF record. When I enabled IPv6, I discovered that I couldn't get through to Comcast users without DMARC, and I also had to request my own /64 from the VPS provider, since the default single address supplied was part of a blacklisted /64 thanks to others. Unless you can get a squeaky-clean /64, forget about it.

SPF has another problem. I sent an email to someone recently @theirdomain.com. I subsequently saw by chance that the email was rejected because they were hosting @theirdomain.com with a random ISP but they had configured the mail to be forwarded to a mailbox in @gmail.com.

Gmail sees the email coming from @theirdomain.com's servers, rather than my server. Gmail checks the SPF record which doesn't match, and it rejects it.

I understand that this style of forwarding is anyway bad because gmail see's all email the user receives @theirdomain.com as coming from those servers, not their true origin. If @theirdomain.com receives (and forwards) any spam, it looks like a spammer to gmail.

Gmail needs to adjust to reality, not vice versa.

The solution to this is SRS[0], but yes, not every mail-forwarder implements this (correctly), and it seems to break DKIM. The good news is that an SPF soft-fail and a DKIM pass usually means a DMARC pass. More information here[1].

0. http://www.openspf.org/SRS

1. https://blog.fastmail.com/2016/12/24/spf-dkim-dmarc/

Even with SRS, the "reputation" of the "middleman" (from Google's perspective) still suffers.

If the mailing list changes the From address, then the message won't show the correct author in the message list in my email client. Replying may work (because of Reply-To:), but not having the correct "From" in the client is really a deal breaker.

Besides, DKIM already solved this problem by not caring where the mail comes from, just that the message was signed. It seems the solution to me is to stop using SPF if you want to use mailing lists.

Which email client is that? Presumably, this could be fixed (well, worked around) there.

Thunderbird. I suspect most clients operate this way.

File a bug? Maybe there's an extension for it, too. AFAIK almost every mailing list has to deal with the DMARC problem by rewriting From.

Why not? The from: address would be the mailing list address, but the from: name could be the original poster's name.

The problem is that address books pick up stuff out of "From: User <user@domain>". So if you now start sending mail with "From: User <no-reply@mailing-list>" you create a lot of chaos.

Some clients have a "reply to list" button, so they can recognise a mailing list. Perhaps they could then also handle the name problem.

DMARC policy rejects messages only when they failed both SPF and DKIM checks. It seems problem is that mailing lists like to mangle message content.

That's not a problem. Mailing lists have been doing that since long before DMARC. Having the mailing list name in the Subject is very handy. Footers are also more convenient that having to look in headers.

Not exactly, the DMARC policy is very flexible. Each server, after a period of monitoring when he receives a detailed report of what would happen if a policy is enforced, can define his own policies.

Ran into this problem setting up a new mailing list system. I'd always wondered by mailing list mail sent from yahoo went directly to spam, while mail from gmail worked just fine.

Mailing list mail will always fail DMARC if you keep the original sender's address, but when setting up DMARC for a domain, you specify what should happen to mail from your domain if it fails the DMARC check.

For gmail and other sane providers, their DMARC failure is set to ignore, which means DMARC failing for those addresses gets calculated into spam probability, but isn't an auto-fail. For yahoo and AOL, they have it set to reject, which means mailing list mail from those domains automatically goes to spam for gmail.

Groupserver mailing list software resolves this by checking the DMARC settings for the sender. If DMARC is set to ignore, the sender address is kept, because it won't actually affect delivery. If it's set to reject, it mangles the sender address so that the mail at least gets delivered.

I've had some contact with a big client e-mail filter, (30k+ users) and we chose to expose a soft SPF in the external DNS and a hard SPF in an internal DNS used as a cache by the mailfilter.

This was a workaround because the big org had countless 3rd party suppliers of services who many times wanted to mail as @big.corp. And this was fine when mailing to big corp as we could whitelist their senders in the SPF service config, but when mailing to the rest of the internet like gmail, yahoo and hotmail we needed a soft SPF fail instead of having to add those countless servers in our DNS record.

Unfortunately, having the list software create a new message breaks threading. It is common practice to Cc people (subscribed to the list or not) that may need to interact with the thread, but they get the message directly from the sender while list recipients get a different message (different From, different Message-ID). Similarly, if the original sender replies to their own outgoing message, it will not thread for list recipients.

Threading can be preserved when creating a new message as long as the mailing software re-uses the same: 'message-id', 'references', 'in-reply-to' and 'thread-index' headers.

Once, I needed to email something to djb. I tried, and his email system sent back a challenge which was dumped into my spam box. I never did manage to get an email to him.

DMARC sucks.

Source: I run two big mailman installations.

Mailman sucks, too.

Source: I administered a variety of Mailman-driven lists for years. Never again.

This is a terrible article that's almost completely wrong as it doesn't even mention the distinction between envelope (SMTP) from and header (RFC822) from.

The envelope from is what is transmitted in SMTP commands and specifies where bounces due to delivery delays or failures are supposed to be sent, the header from is what is displayed to the recipient as the sender, but is completely ignored by SMTP.

SPF only deals with the envelope from, DKIM only deals with the header from (and other parts of the email headers/content).

Really, there is nothing there that necessarily prevents mailing lists from working just fine:

A mailing list can (and should) replace the envelope from with its own address, so that bounces from subscribers aren't sent to the author of the message, but to the mailing list software, which then can do bounce management, such as automatically unsubscribing addresses that consistently bounce because they don't exist anymore. As the mailing list software should use its own domain for the bounce addresses, the mailing list operator can set up SPF to authorize the mailing list server as an outbound server for that domain just fine, and that has no effect whatsoever on the header from that's shown to the recipient, and that replies would go to by default.

As for DKIM, you simply should not modify the message, and it will deliver just fine, whether through a mailing list or directly. Modifying the message mostly really shouldn't be necessary. Reply-to mangling is a bad idea anyhow, and the mailing list should be recognized by the client software either because it's in the destination headers, or by using mailing-list headers added by the mailing list software, instead of mangling the body or the subject.

I found this article from the Fastmail folks to be rather enlightening, and being Fastmail, you know it's meticulously correct: https://blog.fastmail.com/2016/12/24/spf-dkim-dmarc/

Actually, DMARC refines SPF to also apply to the header from. If you have only an SPF record on your domain then the old rule applies (only envelop from). If you also have the _dmarc.<domain> then SPF also applies to header from.

Modifying the message mostly really shouldn't be necessary.

unsubscribe

List-Unsubscribe you mean

> As the mailing list software should use its own domain for the bounce addresses, the mailing list operator can set up SPF to authorize the mailing list server as an outbound server for that domain just fine

This will make SPF pass but it has no effect on the DMARC result. For DMARC to use the SPF result, the envelope FROM needs to be aligned with the header From. Pretty much every mailing list already does what you've described but it doesn't help with DMARC.

> As for DKIM, you simply should not modify the message

Many mailing lists modify every message to add a footer with unsubscribe information. Others only modify some messages when necessary such as to convert HTML messages to plain-text, to strip attachments, to wrap lines to 72 characters etc.

Yes, modify the message and submit to the SMTP server, which calculates the DKIM signature.

I don't understand what you mean.

If you're saying the mailing list SMTP server could sign the message with their own DKIM key, that doesn't work because the DKIM result only gets used if the signing domain is aligned with the domain in the From header.

Ooops, I probably should have quoted you

> Many mailing lists modify every message to add a footer with unsubscribe information.

That will not affect DKIM. The DKIM signing occurs after the listserv creates the email message and hands it off to the SMTP server.

> That will not affect DKIM.

It will affect DKIM and it does.

> The DKIM signing occurs after the listserv creates the email message and hands it off to the SMTP server.

The DKIM signing occurs before the listserv even receives the message. It's done by the sender's SMTP server before it gets relayed. The listserv receives the signed message and adds a footer which breaks the signature.

The listserv has to copy any incoming email message and send one copy of the inbound message to each recipient on the listserv list. To assure deliverability, it has to send it from it's own address @ it's own domain, to avoid SPF failure. Each individual copy is passed by the listserv to the SMTP server to go to each recipient on the listserv list. The SMTP server calculates the DKIM on each message as it is sent.

Again, mail servers adding links is completely irrelevant to the DKIM signing process.

> it has to send it from it's own address @ it's own domain, to avoid SPF failure.

To avoid SPF failure it uses its own domain in the MAIL FROM envelope address. There are very few lists that change the From header, I can't name a single one although they probably do exist.

> The SMTP server calculates the DKIM on each message as it is sent.

Like I said, even if they did, that only works if they were also changing the From header to their own domain. Otherwise, even though there is a valid DKIM signature there is no alignment with the From header so it is ignored when evaluating DMARC. DKIM does not use the MAIL FROM address.

> Again, mail servers adding links is completely irrelevant to the DKIM signing process.

I send a mail to a mailing list, LKML is a good example. My SMTP server signs the message with my DKIM private key and relays it to the LKML SMTP server. LKML receives it, appends a footer and relays it to all subscribers, with my name and email address in the From header but using its own address in the MAIL FROM address. The footer breaks the DKIM signature because when my message was signed that footer wasn't there. LKML does not re-sign the outgoing messages, it relays the DKIM signature provided in the original message. This is standard of many, possibly even most, mailing lists.

Reply on news.ycombinator.com