A deep dive into email deliverability in 2024 | Hacker News Reader

A deep dive into email deliverability in 2024

xomedia.io

350 pointsxoneill227 comments

Given how much weight “Gmail”, “Outlook”, and “Yahoo” email providers pull, I have always wondered about a different type of attack on business entities: “targeted failed deliverability”

Basically in this attack, a victim (particularly a business or mailing list or NGO) is sending out bulk emails to which the attacker owns. Even sourcing this out to shady off shore click farms would work too.

Attacker then marks the victim’s emails as spam in Gmail/Yahoo/Outlook. The “AI spam filters” pick up on this new “spam activity” and will then mark future emails as spam or even delete them before reaching real customers.

After a year, company bleeds money on a quarterly basis. Ad departments wonder why there is decreased engagement through email. Technical departments are bamboozled.

Maybe a big company will be able to weather the storm or just ditch email altogether. But small companies would definitely take a hit. Even smaller NGO or political mailing lists would lose donations (assuming email was a significant source of new donations).

Probably a very low vector of attack tbh, but something that has lingered in my mind.

A webforum I know has a rule against marking email notifications they send at spam (You can opt out of receiving them through the site, they just don't want you doing it on the email client end) to avoid this happening to them. For a small org, it's kind of a real risk?

I'm not sure how larger orgs mitigate it.

They pay 3rd parties to handle relay duties. A large chunk of reliable delivery is based on those received headers, aka what systems your mail is relayed through.

Lol wait, that sounds like e-mail deliverability is almost set up like cartels.

Cartel is maybe overly pejorative, but it's definitely based on relationships.

Postmasters at large ESPs and inbox providers can and will text each other to resolve issues.

This is pretty much how the internet works as well (BGP, etc). It's opaque, but open.

But is the Internet really as centralized as Email has become? I was living under the impression that Internet backbone providers where more diversified than that.

With BGP, you are usually only peering with a couple entities unless you are yourself in the "big boys club". It's a relationship business, but you only need a relationship with the handful of engineers who are on the other side of your own links.

Mass sending is controlled by sendgrid and the likes of that.

Sendgrid is no silver bullet for deliverability. They have many IP addresses listed in blocklists. https://check.spamhaus.org/sbl/listings/sendgrid.com/

Curiously my domain shows as zero but it's part of at least one RBL.

Is there any way for them to enforce that

I run webforums, and this is enforced trivially.

I don't send marketing, I only send transactional emails. Notifications are opt-in, and transactional. Logins / signups are done via sending a TOTP to the email, these too are transactional.

If someone marks notifications of new posts in a subscribed thread as spam... fine, but this is self-solving, as this person will trigger no more email ever reaching them for that email address, meaning they also cannot now sign in to the website, and therefore cannot subscribe to more email updates, or visit things to refresh "last viewed", and so would never be notified again.

Emailing a TOTP as a login has increased deliverability by self-selecting the removal of those who hit the spam button. Deliverability of email from the webforums is over 99%.

Reporting email as spam, effectively bans oneself from the website. I didn't even need to do anything.

When you report something as spam to Google they only claim they "might" cause later emails from the same sender to go to your spam folder; regardless, those emails going to one's spam folder doesn't mean they don't get them at all, it just means they have to look at their spam folder to log in.

When you use services like Sendgrid, having a spam report will automatically prevent future emails being deliverable, third parties and intermediaries do this precisely to protect their email reputation.

Marking spam may not be immediately be a death knell within Gmail, Hotmail, etc, but because of the potential impact from being marked as spam, virtually everything treats it like a death knell.

This is fine for me, my service is entirely opt-in, and if someone hits the spam button it risks impacting other users who _want_ the email, so I am not bothered that this person effectively unsubbed themselves and killed their account.

Surely it's not that trivial given

- Gmail might be able to detect what's important and not mark it as spam

- If it gets marked as spam anyway, the user can just open the spam folder to use it

- Even if you have high deliverability, that might not translate to it actually getting into the inbox

I can only share my own numbers: 90K sent, 97.12% delivered, 103.11% clicked (presumably some clicked more than once? or some older emails still being clicked?)

I do not track "opened" / "viewed" as that is not important to me, these are transactional emails, logins or "someone has replied to your conversation".

am I understanding this correctly: When you mark a single email as spam, you will never receive another mail from that sender?

I run a small org (freelancer) and I have a super advanced bullet proof set of solutions against this. Any 1 of those should do the trick (I do all of these):

1. Don't trick people into signing up for mailinglists.

2. Don't spam.

3. Don't use mailinglists.

My small business is fine.

Not against a targeted attack like the one described.

A clickfarm marking password reset emails as spam could create real harm.

Even a malicious individual doing this to 2-3 fair transactional emails could cause damage for a small business with low volume.

The attack described could not work with a strict DMARC policy.

However of course, most small companies have no idea how to do SPF/DKIM/DMARC.

Could you explain how DMARC would prevent this?

If I attempted to create dozens of accounts on a website, trigger password resets emails, and report them as spam, where would DMARC prevent me?

Nobody is spoofing anything, DMARC won't help. This is marking legitimate emails as spam at scale to trigger actual spam controls against the legitimate organization.

That's just email working as intended. You pay a third party a small monthly fee to handle email blasts via a relay.

Yea I've thought about this but not from the "attack on entities" angle but moreso a consumer-rights / boycott angle. I've had a negative enough experience with a large "maximizing shareholder value" company that I went back through my email history and marked every single one of their comms as spam.

Might be a drop in the bucket, but it doesn't take many votes to make a difference in the spam world.

I'm sure this will evolve soon enough and email delivery might increasingly become pay-to-play with all sort of backroom agreements, if it isn't already.

I bought my dad a sweater for our local MLB team. I made the mistake of using my real email. Ever since I’ve gotten a steady drumbeat of marketing emails and other low value content from them.

Spammers want us to think there’s a significant difference between their newsletter or marketing notes we may have technically signed up for (certainly not willingly) and I don’t feel bad about reporting both of them. If this forces spammers to consider whether recipients will want their messages, good.

It sounds like you absolutely did sign up for the emails, though.

I'm not sure how you could "unwillingly technically sign up" for something like that, especially at the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.

> It sounds like you absolutely did sign up for the emails, though.

Did he? The anecdote here is probably observed by everyone on this forum. How odd that you find it's unlikely to receive spam from a business transaction.

> I'm not sure how you could "unwillingly technically sign up" for something like that

Have you tried using the internet?

> the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.

I would love to live in your world where there's little likelihood of getting spammed just for purchasing something once. Unfortunately, spamming people has effectively zero risk and all reward. If there were any real risk then we would see actual real and frequent consequences every day. We don't see that, but we do see lots of spam in our inboxes.

I do work in digital marketing and there easily could be an automatic sign-up to email without ever opting in. There are plenty of services that will "restart" your email or that will identify you and send emails without consent [1]. If someone is saying they didn't subscribe I would believe them.

[1] https://www.wunderkind.co/how-it-works/performance-marketing...

There are many dark patterns that encourage people to accept their address going on the general spam list:

* wording on check-boxes being sufficiently ambiguous that yo have to think to get the right meaning

* check-boxes on forms in different stages of the ordering process that have inverted meaning from each other

* resetting check-boxes to the “please send me junk” state if the user goes back to correct some other issue or otherwise update the order

* wording the checkbox text to imply that opting out also opts out of getting order update emails (or in some cases opting out does opt out of order updates messages too, and you have to check via some other method what the state of your order is)

And so on.

It is easy to avoid all these accidental opt-ins individually, but it is also very easy to miss one in amongst the sea of dark patterns and one is all that it takes. You need to be vigilant all the time, they scummy marketers only have to get lucky once.

Furthermore, I'm sure some just ignore your opt-out and send crap anyway. When taken to task they'll say you must have accidentally [un]checked the wrong box (how do you prove otherwise?) and tell you that you can opt out from future emails by following the link in the messages.

This is Google's business model, they throw completely legitimate emails your business sends into spam/marketing, so you're forced to pay them for gmail ads.

If your email can be displayed in a gmail ad, then it is spam/marketing.

Yes, that is what I'm saying so I already agree with you.

My point is Google takes content from Column A and puts in in Column B and collects money in their little toll booth. Google pretends to care about spam only when they can't make any money from it, otherwise they don't give a shit unless its a PR issue. Google ads have long been a malware vector - but they don't care because they make tons of cash from it.

I'm not taking the side of either the business or the user here, but this is a forum for start-ups who often have to send out legitimate business emails - which are not necessarily spam.

> but this is a forum for start-ups who often have to send out legitimate business emails

If those "legitimate business emails" can be put as Gmail ads, then they're not legit business emails, they're spam.

Do you have a source that backs up your claim? If that is really the case, then I would share that information more widely.

You don't typically use your main email system for bulk sending, you use a third party for that who is used to taking that heat.

I don't think this would work in practice. My employer sends daily deal emails without using a third-party service like SendGrid or SES and what we do is pay a company like Validity who interface with the big email providers for us. They have honeypot emails that get and validate our emails, they get feedback from the providers on how much they like/dislike us, and we get reports on this.

So an attack like this would be very obvious very quickly, even leaving aside that we'd notice a huge spike in email sign-ups and probably kill their accounts (especially since they're not going to be buying anything from those sock puppet accounts!).

You’re assuming most small companies are as vigilant as your company

If email is business critical, you do it. I work for a pretty small company and we do stuff like this. We have a sender tool that gives us a static sender IP, reports deliverability, click rates and at least estimates open rates. We also have a tool to estimate the quality of a newsletter sign up email address and not collect any disposable emails.

A small company will use a service like SendGrid, or even MailChimp at the higher level.

Those services will do a lot of checking for you and investigating this kind of weirdness/attack is exactly why you pay them money!

If they're sending directly, then they'll definitely be as vigilant as us, or they'd never get emails into people's inboxes!

If they're not using a third-party service what are they using? Daily deal sounds like it's very high volume...

We run our own mail servers. I would not recommend it for most people!

That’s why most e-commerce sites send their marketing stuff from a different domain. If it gets flagged they can still send transactional e-mails from their main domain. Assuming that both mail servers are on different ips.

This seems kinda deluded given that spam prevention teams have been identifying domain clusters for sender reputation management since at least 1998 to my earliest direct knowledge and probably earlier. Maybe it works for a little while, but don't bet your company on it.

That's not spam though. It's promotional material sent to subscribed users.

This is dishonest.

Why? You send to subscribed users with links to your main site. There's nothing fishy in the concept. You're not trying to fool anyone.

I'd argue transactional email should also be sent on a subdomain, to allow for a migration if that subdomain gets "burnt". I'd also argue that really nothing should send email from your root domain at all for this exact reason, but legacy environments can often make this a non-starter.

The problem is then the reputation to humans reviewing links. A legitimate company sending from legitimate-company-transactional-emails.com looks exactly like a phishing scam sending from legitimate-company-scam-emails.com. This can be mitigated by not putting anything important in the email ("You have a new message on MyChart, figure out on your own how to view it."), but people will be confused and it will waste support time.

It's honestly really bad no matter which route you choose, which is why everyone is so intent on getting you into their mobile app that can spam you without affecting their reputation.

Also, even if marketing emails are sent from a separate domain, sales people can be sending so much "spam" that your normal humans-at-the-company emails start getting rejected. I think this might have happened at my last startup; email reputation got bad, interview candidates didn't get their interview conformations anymore. But I might be misremembering.

Subdomains are basically restarting your domain reputation from scratch. I would use the apex domain and a validator in most cases, as long as you're doing the right thing and maintaining list health. Obviously mega-corps might be different and have needs for subdomains.

Isn't that what DMARC policy would prevent? If the emails being sent by the attacker are failing SPF/DKIM, then we can configure the DMARC policy so that Gmail never delivers those fake emails in the first place. So that attack would not be happening.

The attacker isn't sending mail spoofed from the domain--they're intentionally signing up for legitimate newsletters from that domain using a ton of email accounts they control, then reporting those newsletters as spam in each email account.

This could be a pretty interesting use case for a botnet as well, compromise computers, compromise email accounts, for the sole purpose of selling the ability to mass-spam your competitors. Obviously immoral but simply from a technical standpoint I'm wondering if it would work and what scale you'd need for it to be effective.

Moral of the story: Don't send bulk mail to arbitrary lists of email addresses.

> political mailing lists would lose donations

Now you have me rooting for the bad guys.

If political candidates can’t get small donations then that shifts more power to large donations 1 wealthy companies, people, and unions (maybe not the last in the US)

Unions in the US donate hundreds of millions of dollars a year to political candidates.

…and that money has no discernible impact on the voting habits of their members.

...because union members overwhelmingly already vote for the candidates their unions are donating money to? Not sure what point you're trying to make.

Mostly that Unions went hard on endorsing Biden, but their members (at least in the industrial Midwest) dutifully lined up to pull the lever for Trump in 2020.

You know what drives me not to be involved? Knowing that buying a mug from a candidate I like virtually guarantees a torrent of emails from every downticket race in markets I’ve never lived in. And because of the way they set up the lists, unsubscribing means unsubscribing only from “Kelleher for Coroner” in a county halfway across the country. The worst part is the incredible entitlement you experience if you mention this to someone involved in a campaign. I continue to vote, but I haven’t donated a penny to a campaign in more than a decade, specifically to avoid the harassment.

Edit: USian speaking

I know of victims who had their legit email template lifted by actual slammers. The spammers would embed the legit template invisibly in their emails and then only have a few short lines visible with the actual scam. The idea being that filters would see the majority of the email is legit looking and let it in. Eventually users would flag enough of these as spam and the template itself would trigger the blocks. Then the spammers would move on to the next victim who's email template still gets through filters.

Meanwhile the first victim is left to pick up the mess where none of their email gets through anywhere.

Do political mailing lists get through in the first place? I don't live in the US, but someone must've sold one of my throwaway gmail addresses thinking I did.

Every couple of months I end up checking its spam folder and it's just a daily barrage of spam from both DNC and RNC, 1-2 emails per day like clockwork. None of them ever got through to the inbox though.

This change was necessary and long overdue. Requiring domain owners who send significant volumes of email to properly sign their messages allows receivers to more clearly delineate good from bad based on domain reputation rather than IP address reputation.

As more domains send email through shared IP space on transactional and marketing services, having the ability to attach reputation reliably to the sender domain is incredibly helpful in reducing abuse.

Agreed, this is the upside for sure. Despite frustrations, I'm hoping this cleans things up.

The condition about "significant volumes" is not true.

Google states that the new requirements are mandatory only when you send at least 5000 messages per day.

This is a lie. I send at most a few messages per day and usually less than one per day was towards a gmail account and I had implemented a part of the requirements, but not all of them.

Nevertheless, Google has started to reject my messages, so I was forced to waste time with the implementation of all requirements, even if they are somewhat redundant.

Maybe someone else sent 5000 mails to Gmail users using your unauthenticated domain.

http://139.177.194.177/bah.png

That would not have been possible, with the already existing checks.

While the individual messages were not yet signed, Gmail should have already rejected any message claiming to be from my domain that was not sent by my own server.

My domain also sends no more than 10 messages a day. The domain is correctly-configured with SPF, DKIM and DMARC.

At the beginning of March, I started getting temporary rejections from gmail. Not all of my outgoing messages, maybe 1 in 10. Most of these messages were to one individual, to whom I've been sending for years. My domain has existed continously since 2002, and has never sent spam. The rejection message was startling: words to the effect of "Your message has been rejected because of the awful reputation of the sending domain". The reputation of my domain is spotless, according to various testing tools.

There have been no new rejections in the last two weeks.

According to TFA, Google started rejecting a proportion of mail from bulk senders in February. I wonder if I got caught up in some half-baked roll-out of this new (old) policy.

Same here with 20+ years old mail service on the same domain that has never sent spam with correctly configured DNS SPF DKIM DMARC, getting gmail rejections. I noticed a significant improvement after linking the domain to my google account https://support.google.com/mail/answer/9981691

Did you link your domain to Google Webmaster tools? It will let you know what your domain reputation is.

No, I did not. I barely ever use Google services of any kind.

My "domain reputation" should mean something like "the consensus of multiple reputation services", where those reputation services are reasonably open about how they score. It shouldn't mean "the completely opaque opinion of a single, hyper-dominant, secretive provider with all kinds of conflicts of interest".

I can confirm this, I have a mail server with extremely low, non-bulk mail traffic and without SPF/DKIM/DMARC nothing works. Even with, Gmail started rejecting mail (probably noisy neighbor), so the only way around that was to use the little-advertized relay service in the paid Google Workspace when sending towards gmail or workspace domains.

It also says that you only need to hit this limit once and you’re forever on that list. Also domain is considered as the canonical domain, the part right before the TLD. Subdomains are counted together with the apex domain.

The 5,000 message limit adds _extra_ requirements for senders; ALL senders, regardless of volume, must adhere to these standards:

Starting February 1, 2024, all email senders who send email to Gmail accounts must meet the requirements in this section.

- Set up SPF or DKIM email authentication for your sending domains.

- Ensure that sending domains or IPs have valid forward and reverse DNS records, also referred to as PTR records. Learn more

- Use a TLS connection for transmitting email. For steps to set up TLS in Google Workspace, visit Require a secure connection for email.

- Keep spam rates reported in Postmaster Tools below 0.3%. Learn more about spam rates

- Format messages according to the Internet Message Format standard, RFC 5322.

- Don’t impersonate Gmail From: headers. Gmail will begin using a DMARC quarantine enforcement policy, and impersonating Gmail From: headers might impact your email delivery.

- If you manage a forwarding service, including mailing lists or inbound gateways, add ARC headers to outgoing email. ARC headers indicate the message was forwarded and identify you as the forwarder. Mailing list senders should also add a List-id: header, which specifies the mailing list, to outgoing messages.

https://apps.google.com/supportwidget/articlehome?hl=en&arti...

As a sys admin, what do you do when you see 5% of your email hitting spam because the recipient’s Office365 mail server is misconfigured?

Agreed it’s a net positive, but it kills me when the reason emails land in spam is misconfiguration at the recipient’s end. (Like forwarding emails which breaks SPF)

As long as DKIM is configured correctly, it shouldn't matter.

My employer has had massive issues with this recently. Lots of office chatter from HR about needing to call candidates b/c the response rate is down this year due to spam folders eating the offers.

We also slowly but surely are moving towards IPv6 which will make reputation based on IP somewhat useless when I can have as many new IP addresses as I want. They would have to make all newly seen IPv6 addresses not trusted by default when bad actor could send each email from different IPv6 address.

Spam is indistinguishable from malicious content.

You did not sign up for the "newsletter". Your email address was harvested and given to malicious actors hell-bent on screwing you. Clicking on anything will take you to a website where your best interest is not at all what the company is going to do with your information. At best you might just remove one source of junk in your inbox. At worst, you end up clicking on something that turns out to install malware on your machine.

So what should you do?

1. Don't click on unsubscribe links.

2. Click the spam report button

3. Stop using big email services that ignore spam reports. Gmail panders to other big businesses by letting them spam you without giving you the option to blacklist the entire domain yourself. Malicious content will continue to enter your inbox until you move to an email provider that takes your privacy and security seriously.

Fair assessment. Which email service provider do you use?

I use my own email service.

I'm surprised how many big companies fail the one-click unsubscribe test. Whether it's Cloudflare or Akamai blocking the connection, pages that take 5+ seconds to load, pages that require you to sign in or input your email address again... don't be surprised when customers reach for the Report Spam button instead.

I'm using NextDNS with AdBlock list, which is effectively a Pi-hole on the cloud.

The most annoying this is when email senders use click tracking on domains that are blocked by those AdBlock lists. I keep a separate browser instance to copy-paste those links into, but then I have to login again.

I prefer sending unsubscribe emails instead of clicking links. Gmail can automate it.

The only reason some clicks still get through is because they’re using CNAME cloaking to mask the tracker.

Any email I get with click tracking, gets reported as spam.

I don't unsubscribe from emails I haven't opted in to. So report spam it is.

Same. If I subscribed to it (rare), I'll unsubscribe. If I get unsolicited mail of any kind, I'm not "unsubscribing", I'm nuking from orbit by any means available, including reporting to hosting providers. (This has, on occasion, resulted in the termination of a spammer's account, but the success rate is low.)

I'm mostly talking about those emails with helpful hints and upsells that you get when signing up for a new service. People who make a living from spamming^H^H^Hbulk sending may consider those legitimate.

Agreed, I think of it was a simple UI competition. Customers will do whichever is must convenient: unsubscribe or report spam.

I’ll unsubscribe, but now call BS on the “this may take 14 days to take effect…” nonsense in 2024. If I’m getting more emails in a couple of days, they’re getting marked as spam. (Looking at you, TripAdvisor. If you can figure out how to build AI-generated itineraries, you can figure out how to not email them.)

One thing the April changes break is forwarding between e-mail services. If you currently forward from say an old university address at foo@school.edu to a personal GMail account at bar@gmail.com that will no longer work. This must be relatively uncommon if the major providers are charging ahead with these changes but it's pretty annoying for the people affected.

Why will it no longer work?

When you forward an email, unless the email forwarder modifies the message content, it should still match the DKIM signature, so it still passes.

I don't know the details but my rough understanding is after forwarding the next hop delivery will fail SPF.

Yeah, with forwarding, I am seeing DKIM still passes but SPF fails.

We are seeing all the unix forwarders setup a decade+ ago are dead in the water now (have to be replaced with mailing list software)

This explains why I have been unable to get my 2FA from Adobe.

That's my understanding as well, but I also understand that the email will pass DMARC if either SPF or DKIM passes, and DKIM will still pass on forwarded email.

That seems really significant, email fowarding won't work anymore?

This isn't a change. SPF has always broken forwarders that don't touch the envelope from address, and that is right and proper. You can still forward mail, but your forwarder must rewrite the return path.

It's is a change in mail service behavior, and arguably either a broken policy or broken spec (not sure and don't really care since the effect is the same).

"In April 2024, Google will start rejecting a percentage of non-compliant email traffic and gradually increase the rejection rate. For example, if 75% of a sender’s traffic meets our requirements, Google will start rejecting a percentage of the remaining 25% of traffic that isn’t compliant."

https://dmarcian.com/yahoo-and-google-dmarc-required/

It's really not a change or any of those. Mail services have behaved this way in some form or way for a while, these specs have worked like this from the start and the policy is only a logical conclusion of it all.

Untrusted forwarders (also sometimes known as open relays) have been heavily frowned upon for a really long time. SPF has never worked for forwarders that don't do rewriting. Those letters have always been negatively looked upon, unauthenticated mail is a remnant of times long gone.

Google's policies about authentication are the bare minimum you should be doing anyways if you care even a little bit about your customers or recipients in general.

This is something I'm still trying to find someone who understands it in detail and can explain it to me.

Because as you say, you break SPF if you don't touch the envelope from. There's SRS to fix this. But if you do that, you break DMARC alignment.

So... something is always broken? It appears to me there's no way to make mail forwarding that works with full SPF/DKIM/DMARC setups.

I'm surprised anyone's been getting through at all without perfectly configured SPD, DKIM, and DMARC. I've had a well configured self-hosted personal email server for years and still struggle to get through sometimes, though it does seem to be getting better.

I've been wrestling with this for years as well. I hope with these guidelines being published / transparent by the big 3, things behave consistently.

> perfectly configured SPD, DKIM, and DMARC

Just having them perfectly configured doesn't mean that the receiving servers will also see it that way.

Microsoft servers are particularly prone to randomly failing perfectly fine dkim setups for no reason whatsoever.

‘Randomly’ when you just happen to not be part of the mail cartel, they just can’t say that.

Because it’s not just about configuration, it’s also about reputation and a low sending volume you are in danger of getting dropped merely out of not being a well known sender

“Prove you’re not a spammer by sending a larger volume of emails.” True and diabolical.

I am surprised people think deliverability consists of configuring SPF (SPD is Seattle police), DKIM and DMARC. Spammers can do exactly same given low entry barrier.

It's important but have very little connection to deliverability in real world.

DKIM hasn’t been necessary in my experience, and is still not necessary according to the new Google guidelines if you use SPF.

The thing that kills me about DMARC is how often is fails with Microsoft specifically. And also with any use case involving the recipient forwarding mail (which breaks SPF alignment)

I want to follow best practices it recently changed p=quarantine to p=none after fear that legitimate emails aren’t passing DMARC despite properly configured DKIM and SPF.

Hell, I would love p=reject but not until recipients fix their incoming mail servers to handle edge cases like email forwarding breaking DMARC

Senders that apply dmarc and want their emails to be forwarded should use dkim. Forwarding a dkim signed message doesn't break dmarc at all.

> how often is fails with Microsoft specifically

This is the most important part. Exchange (due to its history as an X.400 server, not as an SMTP server) does sometimes mangle the message to the point that DKIM simply breaks. This both breaks origin-incoming and forwarded messages.

BTW, Apple also sometimes mangle messages that it fails DKIM, although I do not know why is this the case (as I doubt they use Microsoft Exchange for their mail service).

this is a long standing problem with mailing lists. they are often configured to add a "[...]" prefix to the subject or add a footer, breaking the dkim signature. this leads some more recently updated mailing lists to always rewrite to their own "message from" header, so they control dmarc alignment for their messages.

for incoming email on mailing lists i'm subscribed to, i don't enforce the dmarc policy. i think this is what the parent post hints at. i'm not sure how easy this is to configure with the various mail server software out there. i'm also not aware how you would configure this with sieve scripts (i looked, didn't find it, but it seems like a basic case).

if you're running a mailing list, hoping for all subscribers to not enforce dmarc policy enforcement doesn't seem like a great strategy.

the forwarding case should be easier to keep working.

The worst is when they accept the mail but silently tag it spam and put it some place the intended receipient will never see it. Google's gmail is the worst about this. Corporate email isn't email anymore. It's a walled garden / silo like Facebook.

> Corporate email isn't email anymore. It's a walled garden

Thank god.

Otherwise I'd drown in cold email spam.

edit: But on a serious note — I'm using Gmail for all companies because I gave up trying to run and administer our own server. It's a travesty that this has become so hard. I feel if you're not on a well-configured Gmail Workspace there's no chance your email gets through, even if legit.

I'm certainly not saying it's right - but it works.

Gmail's spam filter (and promotions filter) works with >99% reliability as a user, with really trivial numbers of false positives.

I haven't found this to be true at all. I've probably marked thousands of linkedin messages as spam, but they still land in my inbox occasionally. I also get random e-commerce related spam from sites I've never heard of. The false negative rate is massive on corporate spam that should be trivial to classify I'd think. e.g. some of them literally have some variation of "this is an advertisement" along with unsubscribe links.

> I've probably marked thousands of linkedin messages as spam, but they still land in my inbox occasionally.

at this point maybe you should just create a filter rule

Cannot confirm. We get a large number of false positives at our business GMail:

Customer requests for quotes, Paypal/Stripe security messages, lots of other important emails go to spam.

See e.g. https://github.com/nh2/gmail-spamfilters-paypal-security-mes...

For a while now I suspect that GMail has some bug with its own group email addresses:

When somebody sends an email to our GMail group email address team@example.com, it shows up in GMail as "Sombody via team@example.com". Of course, teamexample.com receives both spam and non-spam.

I suspect that when we mark spam that comes "via team@example.com", GMail learns that as "things 'via team@example.com' are often Spam", even though info@example.com is a Google Groups email.

And by now, everything that comes 'via team@example.com' is marked as Spam.

So it seems that when we mark an email that arrived at team@example.com as Spam, Google punishes its own Group email address, instead of the sender.

For that case you would have to mark the mail as spam in the groups interface, not via the forwarded mail.

While a possible explanation, that sounds extremely inconvenient, and a silly default.

It's Google's own group of my own org. Does it make sense that the spam button would punish my group own email?

Also, if I mark it spam in the Groups interface, it's still polluting the inboxes of all my team members where it has already arrived, right?

I'd have to tell my team to never use the spam button on obviously spam emails. That doesn't make sense for a business email product!

I think you can set it to be the other way around and have it be "never mark as spam when via team@example.com" - of course, depending on how much spam it gets that might be worse.

indeed very annoying. my understanding is they don't want to give spammers any signal whether their messages was recognized as ham or spam.

i would love some insights in how smart spammers are in actually leveraging such information. most spam seems to be hammering attempts that don't take failure feedback into account.

in my mail server, messages classified as junk keep getting temporarily rejected with a generic error message. at least the (legitimate misclassified) sender gets a delayed dsn, and finally feedback that a message wasn't received.

it seems many mail servers/services think it's more important to not give a signal to spammers than it is to give a useful signal to legitimate but misclassified users. perhaps they think their classification is really great and doesn't misclassify...

Unfortunately a lot of legal protections for unsolicited spam only apply to consumer usage. For b2b, every marketer with the ability to know your email address is entitled to send you as many messages as they want.

If it wasnt walled, it would be completely unusable.

> If it wasnt walled, it would be completely unusable.

It sounds like stiffer penalties are necessary for sending spam, both to consumers and to businesses.

I'm a big fan of proof of work for this usecase, personally.

Exactly how will proof of work help this at all?

I run my own mailserver. I do a catch-all so I receive all email sent to my domain. Even with this setup it is not "completely unusable" in fact it is very very simple to manage spam (entirely in thunderbird, no mailserver side spam filtering) and I have no issues with it. I've had the domain for 20 years now so it is not like it is not out there for spammers to send to.

Sure. Now try it again as an IT manager at a company that buys expensive technology products, with an email address that has been slumming it in lead gen databases for years. Try it again as a loan officer whose job is to trade high-value PDFs with externals whose email address and title is on the bank's 'about us' page.

The ability to know an address is the only license you need to send to it. The only reason you haven't had a bad time with this is because you haven't been a target.

My personal VM has just been placed in some RBL because the entire /24 address space was blacklisted.

Someone (allegedly) sent SPAM and now my machine that sends maybe 3 emails a week is blacklisted

That sucks! And fixing reputation is at best a nightmare. I've seen suggestions about purchasing / sending email from other domains - to protect your primary domain. Not something I really care to do.

i've personally observed an increase in this tactic from both reputable companies' blast communications and marketeers alike.

Google themselves recommends sending different types of emails from different subdomains of the primary domain to help Google differentiate between transactional, marketing, newsletters, outbound, etc

Any half decent marketer will 100% use a different domain for outbound sales (or any use case where spam rate might be abnormally high).

Thank! Will look into this and reconsider.

absolutely. You should/must separate transactional emails (account creations, password resets etc) from EDM (marketing emails).

The "de facto" solution is to outsource that to Google.

Email on a personal machine and domain has been dead for over 10 years.

You just can't own your data.

You can receive it, no problem. But you can't send it.

> The "de facto" solution is o outsource that to google.

So, they have a "de facto" monopoly, or at best, are working with other providers to create a cartel.

> Email on a personal machine and domain has been dead for over 10 years.

Due to the actions of?

> You can receive it, no problem. But you can't send it.

You can send it. They're actively deciding to just block you. Then provide you no recourse.

The monopoly came about "organically". It's not like there was a conspiracy.

> You can send it

Not even that. Nowadays it's not uncommon for some servers to even refuse the connection.

Mind you, I'm not talking about aggressive spammers.

And yes, there's pretty much no recourse.

> It's not like there was a conspiracy.

I'm not nearly as sure. The mechanics of one aren't that hard to imagine. Encourage spammers and make it cheap. Don't ever fight them at their source. Invoke mechanisms that intentionally destroy public utility in public protocols. Force everyone to rely on a small handful of "reputable" senders.

Working backwards from "who decides reputation anyways?" might make it easier to see.

Sending requires quite a large volume for the big players to allow you to play. The only viable option for small servers is to use a SMTP relay service. Amazon's was a pain to get out of the sandbox mode but has been reliable and most importantly free.

As I care more about the recieving side than sending emails this works well enough for me.

yes, this is a pain. the blocklist operators either seem to be not so good at vetting abuse reports, or cause collateral damage to get network operators to take action (while i don't like getting on a blocklist, perhaps it is having a net positive effect for the wider internet?).

i do think mail servers/services are using ip-based blocklists wrong. yes, you can use it as one of many signals. give it some more weight for first-time senders. but if you've been mailing (with spf/dkim/dmarc-authenticated messages/transactions), from an ip that suddenly gets on a blocklist, the previous positive reputation should be stronger than that blocklisting, and you should be able to keep communicating with your known correspondents (until they mark your message as spam after which future deliveries can be rejected/junked). it seems those mail servers/services cheap out and apply ip blocklists early in the smtp process. good for their system load, bad for their analysis performance.

in general, it seems even bigfreemail services are bad at using existing reputation in their ham/spam decisions. i recently switched an online webservice i made (that is about sending certain notification by email) to signups via email (like how you can signup to mailing lists: by sending an email to an address). the idea: if you send my service an email, i'm in your list of known correspondents. so the confirmation reply from my mail server (spf/dkim/dmarc-aligned) should certainly be accepted by yours (you much more opt-in do you want?). i tested with some bigfreemails, and yahoo put my reply with confirmation (that even references the original message) in the junk folder. to people who think you can't compete with bigmail: the bar isn't as high as you may think.

The problem is that your VM provider is (unwittingly) offering the same address space to spammers. Maybe the spammer was using random addresses in the range? Or someone was starting up misconfigured smtp servers permitting relaying.

Running an outbound smtp server on a customer ip-range is going to be problematic anyway. All such ranges can be considered suspicious since the spammers who use them don't care about their standing.

True.

But I've had my IP for years and it was originally clean as well.

I think you mixed envelope (RFC5321) and headers (RFC5322) in your text.

The domain name in the From: field in the email envelop header is inspected and aligned with other domains authenticated by either SPF or DKIM:

The envelope does not have any header, the headers are in the content/body of the email. Also your screenshot of the "Here’s an example email envelop from an organization that passes all of the email security guidelines:" are the mail headers and not the envelope information.

Great presentation on this topic from dmarc.org

https://dmarc.org/presentations/Email-Authentication-Basics-...

Fixed, thanks! Embarrassing :).

Roughly 50% of my daily Spam comes from @gmail & @hotmail/@outlook accounts.

What exactly are they doing about that ?

They may not - but you fail to properly set up DMARC.

at least gmail.com has a dmarc policy p=none, so a failing dmarc check is not a reason to reject email. i don't think it's that common for small-scale installs to enforce dmarc policies. there are other signals to use though.

plenty of bigfreemail spam is actually sent from their network, they're an interesting target for spammers, and at least some of them put a lot of effort in preventing abuse.

> What exactly are they doing about that ?

Nothing at all, because they are too big to care.

Aggressive decentralization is the only way to save the Internet. Host your own email, get everyone you know to host their own email.

Fantasy solution, will never happen.

You might be surprised, if you're young enough, to hear that this is exactly how it worked for a very long time.

At my first corporate job, and my second one as well, all email was sent and received directly from/to each persons individual workstation. There was no concept of centralized email server.

(Let alone outsourcing it to some other company, that would have been completely unthinkable.)

lots of things used to be a thing

Yep. It seems to be from stolen accounts, maybe gathered from leaked account/password lists.

The result is that outgoing hotmail/outlook smtp servers are added to blacklists until they start content filtering their own users.

It’s not even stolen accounts. There is a huge amount of cold sales email (spam) sent specifically through Google Workspace accounts. Google do absolutely nothing about it.

Why would they do something about it? They're getting paid whether you receive the spam or not.

Any regular account - a seat - in Google Workspace is limited to sending 500 email per day. I

I ran my own mail server for more than a decade. Same IP the entire time, never sent spam (for personal use only.) Finally threw in the hat last year and moved to a paid service - it was a pain to tell every person I sent mail to to check their spam box and mark me as not spam or add me to contacts. Beyond that, gmail smtp servers kept getting onto spam blocklists, so I wasn't receiving mail from gmail at times.

Gmail and Microsoft require a certain volume of email hitting their servers otherwise they forget you exist. I ended up switching to using Amazon's SES service to keep my selfhosted server running.

I use seperate emails for each service much like a seperate password so I'm heavily invested in keeping my own server at least for recieving.

Speaking as someone in an industry that receives a lot of unwanted and seemingly un-unsubscribable marketing emails. I have never ever bought anything from a company that has sent me an email cold. I have my inbox set to show the first two lines and I delete them without opening them pretty much all the time. The only thing marketing emails do is annoy me.

So what any competent sysop has been doing for years?

Sometimes it's not about competence but about priority. If email is something a business does as a side thing and not a core thing, and if email keeps working without changing anything, there's no need or priority on setting up newer things. From a priority perspective at least.

But yes if it's a recent setup, or email is a core part of the product, any competent sysadmin should have been doing this.

1. GMail will block your email if you don’t allow one-click unsubscribe. But this is very insecure since anyone can unsubscribe you if you forward your email

Easy Unsubscribe: Implement easy unsubscribe options (One-click Unsubscribe). Gmail users have tools to report spam, unsubscribe from unwanted emails and control their inbox experience. If it is too difficult to unsubscribe from your emails, customers will be more likely to flag your email as spam. Additional links provided in the ‘References’ section at the end of this article.

2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click! So all your emails will be blocked.

https://getcake.com/apples-intelligent-tracking-prevention-2...

Why would apple identify the domain used for unsubscribes as in use solely for being a bounce tracker?

Their heuristics are proprietary dunno

So you argument here is based on something that hasn't happened, but just something you pulled out of nowhere. Or was this meant humorously?

Gmail does not require one-click unsubscribe, what they actually require is that you include the “List-Unsubscribe” header in bulk emails, with a functioning mailto or http target.

If I forward your newsletter, that’s not a bulk email and it won’t include that header.

This is an important distinction that seems to get glossed over in a lot of the coverage and guides about the recent Gmail and Yahoo changes.

I guess if you forward it as content but I'm surprised you don't default always to forwarding as an attachment.

The default forward action in Gmail sends it as content. Forwarding as an attachment is a well-hidden option in the web UI and I just looked in the Gmail mobile app and could not find a way to do it there.

>2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click!

>https://getcake.com/apples-intelligent-tracking-prevention-2...

Your source doesn't actually say that "ITP will start removing all the information from the URL", only that it will "limit it the same way as third-party cookies" and will be "purging website data in such instances".

You can do one-click unsubscribe with headers as well, those aren't usually forwarded by MUAs.

Then there's the other side - receivability. IDrive is supposed to send me an email each day reporting backup status as seen by the backup servers. Those messages have been flaky since mid-February. Logs indicate the backups run; it's just the completion emails that fail.

Their support people blame me, although they admit others have the same problem. They're not using a mail delivery service - the emails come directly from an IDrive server.

They're sending to my web site, which forwards to my personal address. There's no filtering at the first stage, and a division into Accept/Greymail/Junk at the next stage. Neither Google nor Yahoo is involved at any point.

I’ve gotten some emails from Gmail about delaying my emails to Gmail users because I apparently send too many emails. I use git-send-email(1) which might send a cover letter plus X patches right after each other. These Gmail users are then in the CC. So I’m not a mailing list. The email list is the To recipient.

I’ve been wondering if this was the cause. I don’t send out 5000 emails (I’m not 10X). But there’s this part:

> While these guidelines primarily affect bulk senders, senders with less volume per day can also be affected if they are not adhering to these guidelines.

I haven’t looked into it yet but I guess I should.

I use my own domain and I’m hosted by a not-Gmail provider.

https://github.com/trusteddomainproject/OpenARC/issues/163

https://github.com/trusteddomainproject/OpenDKIM/issues/186

OpenARC/OpenDKIM don't parse email headers to spec. Help wanted.

Use rspamd?

thanks, that seems promising. i will look into it.

I wonder how this ends up impacting government agencies and especially courts and law firms. My experience has been all three struggle with these things.

As usual in any email thread about email deliverability, the amount of FUD in these comments is absolutely mind-boggling to me. I’m not unusually smart or intelligent or capable. I wouldn’t consider myself a deliverability expert. It’s only a small small part of my job. I’ve never worked for any organisation that sells email delivery services to third parties. Why the hell can I understand this stuff, and get it to work, while there are so many people here that very clearly indicate (via what they’re saying in their comments) that they DON’T get it yet have a serious axe to grind?

I’m left feeling like homegrown email delivery is some sort of lightning rod for stuck-in-the-past faux-sysadmin types that can’t get past the fact that it’s not 2003 anymore and lazily / maliciously comply with SPF / DKIM.

IT’S NOT THAT HARD.

Without knowing everyone’s domains and IPs and history, it’s hard to judge competency in a thread like this. SPF, DKIM, and DMARC are important to set up correctly, but doing so is NOT sufficient for good deliverability. In fact it is only a small component of success.

Well-established organizations that have a long history of sending steady volumes of high quality content with low complaints have a huge leg up. So if you work at such a place, or you contract with such a vendor, you’re going to feel like it’s obvious that the DNS entries work well.

Yeah it takes like 5 minutes. I have no idea what people are crying about. On a tech site no less. This is entry level have the intern do it stuff.

>As usual in any email thread about email deliverability, the amount of FUD in these comments is absolutely mind-boggling to me. [...]

What type of "FUD" are you talking about? The objections in the thread seem pretty well founded (eg. being shadowbanned despite complying with SPD/DKIM, or this requirement breaking email forwarding), and there aren't really any that are against implementing SPF / DKIM.

I agree on the technical part, but:

A problem is that you can do everything technically right, and sitll land in spam, because some big players don't play by the usual rules.

For example, Microsoft apparently has an allowlist for IPv4 -- or equivalenty, blocks all IPv4 by default, until you manually de-list them at sender.office.com. At least I haven't found an IP yet for which I didn't have to do that (self-hosting email for 15 years).

(Imagine every provider did it like MS; you'd be sitting there and filling out web forms with 1000s of providers.)

So you have a technically perfect setup and MS stil rejects you.

That said, using some provider to send emails for you doesn't solve deliverability either. There, many customers share the same sender IP. If one of them sends marketing/spam, the entire IP gets bad reputation. In such cases, providers recommend to upgrade to "bring your own IP", which then needs to "gather reputation" [0]. Great, might as well have self-hosted in the first place, as repuation is the only thing I bought the service for.

[0]: Example: https://www.mailgun.com/blog/deliverability/dedicated-shared... -- Especially entertaining is "Use a shared IP if: Your shared IP partners have built a good reputation." As if you had any control over that.

> A problem is that you can do everything technically right, and still land in spam, because some big players don't play by the usual rules.

That applies to really all email providers. Part of fighting spam is (somewhat unfortunately) not telling the spammer what you're detecting.

Congratulations for being lucky enough to have big actors accept your e-mails, this is not the case for many of us who do understand and apply SPF, DKIM and DMARC. Guess we'll just try being luckier...?

I have the same gripe and in response published the notes to set up my self hosted email on my blog a few months ago [1]. It's really not that hard but yet we constantly see this topic on HN. I understand there are people who've set it up properly and still have their mail end up in spam. Maybe we're just lucky. But there's no need to write long pieces about this going into detail what this tech giant and that tech giant do. All you can do is set up DMARC, DKIM and SPF - that's it.

EDIT: Admittedly this post is also about bulk sending where other metrics like unsubscribe links and spaminess matter. But for the self hosted crew it really just comes down to DMARC, DKIM and SPF.

[1] https://tschumacher.net/self-host-email/

What I have gleaned from the HN discourse over the years is that the people who are mad about this topic actually are spammers. They want to spam you and they are super pissed off that their little scammy idea isn't working.

This looks like a good document, but the author made it political by referencing "Hilary" Clinton and her emails and linking to some Trump stuff. I can't take tech stuff seriously that's dropping in political crap. Go away!

Yes, this put me off enough that I was no longer interested in reading the article. The silly reference is one thing, but was a link to that YouTube video really necessary?

And it's for a company? OP you might want a part-time PR person to review these...

Removed! Yeah, bad attempt at humor. And certainly not done in support of said candidate.

Very valid point. I just removed my poor attempt at political humor. It was not meant in support of any candidates.

Are you a truth seeker or attention seeker manifesting your political preferences instead of focusing on what article actually discusses?

that's a rhetorical question - you already answered it.

I run an email forwarding service[0] and it's damn hard to get into inbox of any major provider if SPF/DKIM aren't config properly. DMARC or ARC might be optionally but an email without SPF/DKIM, good luck having it hit any inbox.

Office365 is the toughest, email just randomly land on spam no matter what I do. Icloud, actually it's ProofPoint is tough sh*t too.

So I'm so surprise these guide just pop-up now like it's a new thing.

---

[0]: https://mailwip.com

This was my initial thought too. These guidelines have been around for ages, and just now being officially implemented in 2024.

> but an email without SPF/DKIM, good luck having it hit any inbox

And that's a decent thing.

Someone that doesn't do that has no business sending email.

The problem is that even doing that and even sending only a couple of emails a day is still usually considered SPAM.

I’ve been running a personal mail server since 2016, and I’ve been struggling mostly with Microsoft off-and-on. I haven’t make any changes since setting up DMARC, SPK, and DKIM in 2016, but I’ll still sometimes randomly get blacklisted for 2~30 weeks for seemingly no reason, and then get unblocked again for seemingly no reason.

It’s recently started happening with iCloud too. For 5.5 weeks any email I sent would either get bounced or land straight in Junk, until yesterday when the powers that be decided my mail was worth delivering again.

I’ve somehow never had an issue with Gmail or Yahoo, and of course never with any other non-big-three mail servers.

Could be your IP block gets the odd spammer or you don't send enough emails and their servers resets you IP address reputation, meaning you essentially become a new email server.

ProofPoint is indeed disgusting service. Wonder how they proliferated into corporate world so much and so fast.

No mention of BIMI (either for or against)? I'm surprised...

It doesn't affect deliverability in a measurable way compared to DKIM/DMARC.

The article is branded as a "deep dive". If it doesn't affect deliverability in a measurable way, then at least mention it in passing to say that it doesn't affect deliverability in a measurable way...

Valid point, thanks. I debated added a section concerning BIMI :). I'll add something in the next revision.

Just added the BIMI stuff.

Nice! Thanks.

Isn't there any open source project that solves the e-mail delivery problem? If not, why not? This sounds like something that can be fixed by software.

There's some online tools out there, and an OSS project that'll scan DKIM reports for you - but this was all I could find. Beyond this, there's commercial tooling / services. I may implement something that keeps a tab on things if I come across any issues.

for a long time it hasn't been a software problem. It's been a reputation problem, the only recent change is we're more interested in the domain then the IP now.

Please elaborate on this, as I'm not exactly sure what you mean. The email deliverability problem is a side effect of false positives in spam filtering. Unless you have a proposal to completely eradicate false positives?

What do you mean by the email delivery problem? Do you mean setting up DKIM, SPF, and DMARC? It's not that hard, compared to setting up an email server in the first place.

> Isn't there any open source project that solves the e-mail delivery problem?

Aplogies if you do, but it sounds like you misunderstand how email works.

The problems are stricly in policy, not in software. So there is nothing an open source project can do. The problem is that "too big to fail" megacorporations like microsoft just randomly decide to block incoming email from most of the Internet.

If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.

(That all said, I run my own email infrastructure since long ago and it works fine. But I know some people struggle, which is contrary to the intent of the Internet.)

the centralized power doesn't exist, but then you run into every misconfiguration possible. is it better? maybe.

Yes, it is better. The reason is that when every email operator is small, the overall email community can pressure them to go fix things. You start by sending notes to postmaster@... with problem reports. If things go ignored you start blocking them and then they wake up fast.

In today's world when microsoft or gmail or yahoo start misbehaving in email handling, what are you going to do? It is impossible to reach them and while you could block them, they will never notice and you're only hurting yourself.

It is the classic problem of centralized power which hurts everyone except the (near-)monopolists.

> If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.

You would have different problems though. Spam would still be hard, or arguably even harder, because you would have lots of small mail servers with no idea if they are trustworthy or not. Is this uncle Bob, who I havent spoken to for 30 years, sending me a heartfelt message about the family? Or is it a scammer trying to cream some cash out of me?

> Spam would still be hard, or arguably even harder

Doesn't make any difference. The major sources of spam today are places like gmail and outlook servers. You have to run some message classification regardless of where it came from.

> The major sources of spam today are places like gmail and outlook servers.

By volume perhaps, but is that true as a proportion of all mail that was sent via these services? I'd be surprised, though I'd be very interested if there where any numbers publicly available.

> By volume perhaps, but is that true as a proportion of all mail that was sent via these services?

Would you care? I can say that I don't care (as someone running my own email infrastructure).

I run filtering on all incoming email, I don't assign any weight to which email server it came from because it's not a useful factor.

The spam situation could also be better if we had designed into that space. Want to send a single email to a sender that you don't have an existing relationship with? Hashcash. Have a web service that sends lots of emails? Get user consent beforehand with e.g. an oauth token to bypass or lower hashcash difficulty.

I wish we could solve the unsolicited SMS problem.

I have received exactly 2 such messages in the 25+ years I've had a mobile phone. I imagine this is a regional thing, and that where you are eithet mobile networks are insecure, or operators are in cahoots with spammers?

I totally agree, and also unsolicited calls. Right now the only options that I'm aware of are:

1) Blocking numbers (which is pointless because they rotate them and spoof residential numbers).

2) Whitelisting numbers and blocking everyone else (this would cause me to miss legitimate calls).

3) Blocking entire ranges (this doesn't work because of the spoofing).

4) Using one of those spam screening services (currently looking into this). Still a bit concerned about missing valid calls and the privacy issue with this.

Unsolicited calls are annoying but not that much of a problem, because unknown number real people that have a good reason to join me can always leave voicemail, while spammers never do (not worth it for them I guess) ?

I haven't got any since I switched to Android. Same with spam calls. Their crowdsourcing really works.

Just a freindly reminder. I wrote a thorough guide on email hygeine here, includes validation tools to help troubleshoot & straighten things out.

At an even more basic level than what you described here, I recently improved my delivery rate by adding a name to the transactional emails that we send (from: 'My Name <admin@foo.com>') where previously we didn’t have a name. As well as cleaning up subject lines - where previously we included some abbreviations. Providers like mac.com were previously soft bouncing some emails, and now they seem to accepting them.

Hotmail/yahoo/aol all still seem to shove our transactional emails into spam pretty often (judging purely on the amount of those users who fail to confirm their accounts).

Thanks for the info. For years, I've been using trial & error / reverse engineering approaches to improve deliverability. Frustrating!

mac.com is the worst. I ban that domain from new registration now.

And Apple sends no dmarc reports nor do they implement any actionable feedback loop.

Reply on news.ycombinator.com