Hacker News Reader

Comment by stebalien | Hacker News Reader

I used to do this. What finally killed it wasn't reputation, it was the fact that I needed 100% uptime or risk losing messages, getting my address blacklisted, etc. Email is supposed to be resilient to down time (retries, trying each MX record, etc.) but I found that large mail providers tend to just bounce and walk away.

Worse, GitHub (back in 2016 and 2018) would mark a recipient as "unavailable" after a single bounce, refusing to send any more notifications to that address. They since improved the situation and their support was actually very helpful and responsive here, but it's pretty clear that modern SMTP senders have an expectation that recipients will be "always online" that didn't exist when the protocol was invented.

View on HN

I have a feature (called greylisting) whereby my server intentionally rejects the first mail it receives from a domain.

I have never had anyone claim that their mail has not been delivered to me, and I get a lot of mail.

Retry is built in to the spec, and if you’re really worried you can put a second “receive” SMTP server on the internet with a lower priority, and have it backhaul with LMTP.

———

Email was designed in a time where hosts were not perpetually connected to each other.

GMail itself will sometimes temporarily reject messages, then accept them later.

I have Postfix logs showing things like "this address is receiving a high rate of email" which are later accepted.

Gmail always rejects the first email I send to a new gmail account. It does this every time – and has done for years – despite the fact I have sent emails to hundreds of other gmail accounts, and send emails to such accounts every day.

This is the reason I personally will not touch any Google services. And in business, I excise Google services as a priority. If a company cannot handle email in a civil manner, it certainly can't be trusted with anything of importance.

> it was the fact that I needed 100% uptime or risk losing messages

Q: If your server(s) is/are offline for a few hours, why would you "lose messages"?

I've just checked my own email server -> "up 219 days"

Honestly, compared with the stuff we do all day, this is not hard...

> Q: If your server(s) is/are offline for a few hours, why would you "lose messages"?

They said...

>> Email is supposed to be resilient to down time (retries, trying each MX record, etc.) but I found that large mail providers tend to just bounce and walk away.

I take that to mean that if your server isn't availble to receive the mail at the time it is first offered, it won't be retried later. That wasn't the case (for most mail) when I gave up on self hosting 10 years ago, but it's plausible.

It's not reasonable. Mail not deliverable is not the same as house burned down, recipient moved unknown or sth, it simply means the letter was not received. Who and why messed up is unknown, thus NO mail server will mark you down after a single attempt.

Host your own!!

Reasonable and plausible are different things. I wouldn't be surprised if some outgoing servers just never get around to sending retries.

> I take that to mean that if your server isn't availble to receive the mail at the time it is first offered, it won't be retried later.

Umm, RFC 5321, which describes queuing and retry? SMTP is designed to be very forgiving of transient network issues.

> That wasn't the case (for most mail) when I gave up on self hosting 10 years ago, but it's plausible

Plausible? To those of us who run our own mailservers, the OP's statement is an extraordinary claim.

This is fearmongering. My mails always got resent after some hours or a day. It's absolutely NOT possible to tell if the problem is on your side, senders side or somewhere in between why a mail is not delivered once and no standard server config would simply toss it.

Host your own mail. I get 99% deliverability with 0 repuation since i do dkim and spf correct.

Don't be distracted by the "complexity" - if you config right it's totally doable.

Gives you actual private caldav too btw

>I get 99% deliverability with 0 repuation since i do dkim and spf correct.

Your anecdote of success doesn't matter to the others that correctly configured DKIM/SPF and still don't get their emails delivered to Gmail/Outlook/Yahoo/etc. E.g. : https://news.ycombinator.com/item?id=32715437

One of the reasons for hard-to-diagnose sending failures is that Gmail/Outlook have "extra invisible rules" that override correct DKIM/SPF settings because spammers and phishers also have correct DKIM/SPF. So they use extra heuristics such as "ip reputation" etc.

And even after one gets it working, e.g. "submit some form" to Microsoft and wait a few days to get things unblocked... the deliverability may break again because of another "invisible heuristic".

EDIT to reply: >No, that's because your relay overwrites part of the header which makes dkim strict break. Change to relaxed or don't modify the header on your relay.

Delivery reliability can still break without using a relay.

In fact, this unreliability of 100% self-hosting at home is why some self-hosters split it into a hybrid setup and add an external relay for outgoing SMTP and only keep self-hosting for receiving email.

No, that's because your relay overwrites part of the header which makes dkim strict break. Change to relaxed or don't modify the header on your relay.

Outlook business will accept your mail, Outlook private may filter, but the rates fluctuate so heavy i suspect its rules based on user behaiviour/interests. I dono, cant have both spamfree inbox and 0 false positives.

I hate the fact that your comment got flagged / greyed out / whatever even though it's perfectly correct. I'm one of those people who had configured everything perfectly. Score of 100 on mail-tester, SPF, DKIM, DMARC, you name it. Examining the headers in an e-mail sent to gmail: pass, pass, pass. Everything green.

Microsoft however? Denied, 100% of the time. Spam folder, or even plain rejected. Why? No idea, they won't say. They redirect you to their shitty partner that you can PAY in order to HOPE you get approved.

I don't know why our experiences are considered "anecdotes", and not the other way round. What's the incentive for big players to accept e-mail from home servers or small dedicated servers? "Sure it could be Standard Nerd from HN running their own stuff for street cred points, or it could be one of the bazillion spam factories sending fake UPS scams. In doubt, let's reject."

>I hate the fact that your comment got flagged / greyed out / whatever even though it's perfectly correct. [...] I don't know why our experiences are considered "anecdotes", and not the other way round.

It's because people who successfully self-host think their situation universally applies to everyone.

Here's another example from 2017 of someone replying to my previous reasonable comment about self-hosting by overconfidently saying I was exaggerating the issues : https://news.ycombinator.com/item?id=15526127

And then 18 months later in 2019, that same person reveals they also got their sent emails rejected by Gmail : https://news.ycombinator.com/item?id=19757607

So they end up solving it by "outsourcing" the outbound email to a relay (SendGrid).

So my comment gets downvoted for explaining what others had to do in the real world.

The following should not be a controversial statement but for some reason it is: Correctly configuring SPF/DKIM/DMARC and getting 100% green score on https://www.mail-tester.com/ for your self-hosted setup ... does not universally mean your outbound email will get accepted by all the services.

Read the logs from Gmail and Microsoft, they will tell you exactly why the mail was filtered. Act on that problem and have your mail appear in inboxes.

It's usually relaxed DMARC triggering Microsoft. Gmail accepts relaxed.

I add it here so you can successful self-host: You need strict DMARC for Microsoft. If you change the header on your relay DMARC relaxed filters will pass the mail, but not strict.

Because this adds the need to sign every single mail for every single recipient (expensive) its safe to filter for this as a SPAM-Server will sign mail once, then distribute.

That's why your mail is filtered - not because your non-blacklisted IP is the problem or whatever.

>ip reputation

Get this. I owened a /23 for 7 years (still own it today) and kept the mail server ip on a /27 just for the mail server on a /24 that was not used for anything production (firewalled and maybe 3 ip's responded on port 443). My mails were banned for bad reputation. The provider which hosted my /23 was well known for responding to abuse, even falsely flagging my account as abusive in the early days for simply _sending_ valid smtp mails.

IP reputation turned out to mean, if they never saw your IP, you were in the banned bucket. How do you even fight against that

I think i found a loophole for the google and outlook ones... I have had my domains on both providers, and then left to my own (but left a couple of google and ms txt records by mistake) and never had any issues delivering to both providers. Thinking of doing the same thing again honestly, but looking at good providers at the moment.

Until that one email you wish to send to someone important never goes through.

The fact is, big email providers have all the leverage and you will have to play their game ($$$) in order for your email to work everywhere.

It happened to me and that made me realize it's not worth the hassle. Good luck

I know right. It’s like, “what did they do to my boy?” as to huddle over the bullet ridden corpse of your son.

Reply on news.ycombinator.com