Hacker News Reader

Comment by that_courtney | Hacker News Reader

I feel like this solution is optimizing the wrong problem.

The bulk of work with managing a mail server (these days) isn't software setup and admin. On the receiving side, it's all the work dealing with abuse and attacks. On the sending side -- and this is the tough one -- it's getting sites to accept your email. When I finally gave up managing my own mail server (about two years ago), I found that about every six months I was involved in some panic where some large mail provider (Microsoft and Google most frequently) decided they didn't want to accept email from my server. Solving these issues is neither easy nor quick.

These days I'm very happy to pay somebody else to run email services using my provided domains.

View on HN

THIS.

Spam is a solved problem, thanks to SPF and DKIM. But despite doing all the right things, Microsoft and Google continuously block and rate-limit delivery.

Case in point: we deliver 20,000 booking confirmation emails every day, all requested by users and not spam. We have perfect Postmaster Tools metrics: absolutely zero reported spam, 100% IP reputation, high domain reputation, zero feedback loop spam, 100% encryption. But Gmail still rate-limits us daily, so their customers get their booking confirmations several hours late.

It's getting to the point where running an independent mail server is impossible. Yet another part of the open internet bites the dust.

How many servers do you have? Just curious if the rate limit is per IP address?

We have a Microsoft email subscription (all our mail is hosted with them), and their smtp server rate limits how much mail we send to our customers. We can’t even send 30 emails at once. We had to implement retry logic with back off.

Why not just use Sendgrid or similar for SMTP emails to customers, get a dedicated IP with them and have way less concern.

I always prefer to isolate those things from the main corporate email and reputation anyway.

You don't want your main domain getting hacked and affecting customer emails or vice versa.

Have you spoken to an account manager there? I send way more than that on a regular basis. I do it sequentially for simplicity, I wonder if that makes a difference.

You should be using a dedicated transactional email provider, and accept the additional cost. At most scales, this is the correct business decision.

Really hoping ENS can step in here and fill this gap. There's a lot of work being done on social and messaging infrastructure. Using that as an email and chat handle and social handle would be huge for maintaining open messaging. Unfortunately this relies on adoption and people living past their bias and admitting there's use cases.

What is ENS?

ethereum mail service? https://www.skiff.com/blog/skiff-ens-ethereum-mail

Whoops, Ethereum Naming Service

I’ve been in the Ethereum ecosystem for over six years.

I’ve heard “there’s a lot of work being done on…” literally thousands of times. Yet outside of the crypto-sphere I’ve only met a handful of people who’s life would be impacted in the slightest if the entire thing disappeared.

Ethereum is eight years old and debates about use cases, etc aside I think even someone towards the middle of the debate can readily acknowledge progress, utility, value, and adoption has been incredibly slow.

Consider ENS - the last time I registered with ENS the process was just as confusing and convoluted as ever, only in this instance with gas fees I paid around $300 for the name. Oh, and the various transactions kept failing - leaving me wondering (per usual with crypto) if my funds went up in smoke, off to some scammer, etc.

Like so many times in crypto before I just assumed the funds were gone but like I said I DO follow this space and DO find it interesting so I consider burning crypto left and right a relatively cheap education/hobby. Plus I get to tell cool stories like this one on HN ;).

I can’t imagine how creaky, bug ridden, impossible to scale, nearly impossible for mortals to use, and inevitably completely overrun by scammers and frauds an Ethereum (or any crypto) based mail and messaging system would be. No. Just no.

With every day, week, month, and year that passes hearing “We’re building! People are working on XYZ!” when meanwhile the only thing people actually see is the latest headline for some crypto criminal, fraud, scam, collapse of last years “use case” (NFTs anyone?), etc.

I’m not saying it’s dead but I think it’s pretty safe to say the window for anything approaching mass adoption, credibility, and real utility has closed.

“Stop trying to make fetch happen! It’s not going to happen!”

You must only be following at a pretty surface level if you think "things are being worked on" is just a meme

Things are being worked on but nothing changes, crypto is still just a speculative asset for people to try making big bucks quickly with, people use monero to buy drugs. Those are the two usecases, it doesn't matter how much people work on cool crypto things if the usecase stays the same.

100% agree. Been in crypto for 12 years. I’ve held this view for the last 7 years. So much delusion in the space. So much “technological orgasms” with no grounding in real world business dynamics and economics.

Precisely. I follow because there is some interesting technical work being done. There is a lot being built but even after tons of time, effort, and resources (at least 10s of billions of dollars) the only thing that ever comes out of this work is another incredibly dense and nearly impossible to use "technological orgasm" with an extremely thin use case utilized only by (seemingly) the team that built it and a minuscule circle of hardcore enthusiasts.

MetaMask is by far the most popular wallet used for all things Ethereum and Web3. They touted "30 million" MAUs[0] during the peak (3/22) of the last crypto craze/bubble. Let's put that in perspective - there are an estimated 5.3 billion people on the internet. 30 million MAUs represents 0.06% of worldwide internet users. Facebook (which has been in rapid decline) has at least 2.6 billion MAUs. One social media platform vs an entire ecosystem (MetaMask can easily be characterized as the official onramp to Web3). Literally a half of a half of one percent of the TAM (internet users).

Want a (somehow) even bleaker picture? According to DappRadar the top 10 most popular Dapps[0] COMBINED have a lifetime total of 888k wallets (all numbers rounded up) that have ever interacted with the Dapp in any capacity. That's 0.00167% of worldwide internet users and I'm sure the MAUs for these Dapps are even more pathetic. Additionally, it's widely known that wallet != user (many users use multiple wallets and addresses). Looking at these Dapps none of them have any utility other than "DeFi" (swapping tokens around), play to earn games, and straight up gambling. To look at it another way, an absolute best case (impossible) scenario for MAUs of these apps worldwide is roughly the equivalent of the entire population of a small US city. Have you heard of Lee's Summit, Missouri (to pick one)? Yeah me neither and the same could be said for these Dapps.

All of the building in the crypto ecosystem is by crypto zealots for crypto zealots which the MetaMask and Dapp numbers show is an absurdly tiny (so small you can barely wrap your head around it) population. Crypto (other than gambling on exchanges) is an extremely obscure fringe religion.

[0] - https://decrypt.co/95039/metamask-consensys-30-million-users

[1] - https://dappradar.com/rankings

I'm a former sysadmin and as such can remember the days when it was almost considered fun (at least by nerds) to run your own mail server.

Those days are long gone, and the reason why is that monopolistic companies hate open protocols like email and want to 'monetize' everything they can.

How is spam related to SPF and DKIM? Those prevent forgery, but if a spammer actually owns a domain, they can send you whatever they want. That's where the majority of spam comes from, so it's far from a solved problem.

A lot of spammers forge, requiring SPF/DKIM does help a bit. But yeah, not a solution.

SPF and DKIM allow using domain reputation instead of IP reputation. Especially with IPv6 IPs are fairly easy to aquire and can be "stolen" either by spoofing/highjacking the network or more easily by compromising some hosts and sending from there. Hijacking a domain is generally harder as you need to compromise a mail server to get both the IP and DKIM key. And there are less mail servers than WordPress instances to hijack.

Sure, you can buy domains but that leaves some paper trail and generally you will see added spam scores for new domains.

So SPF and DKIM don't improve spam scores on their own, but the give a much more reliable path to building reputation.

Spam is only solved by the large mail providers (who have enough data to act on it within a split second).

After the great Gmail exodus two years ago (when they killed then unkilled legacy domain accounts), I moved to MXRoute — and the amount of unfiltered spam I get is insane. It's a daily nuisance. I have added about 200 filter words now, blocked hundreds of e-mail addresses, but it's next to impossible to filter out sometimes.

From conversations and threads online by the MX Route admin, it seems like this is a nearly unsolvable problem, still. Spam evolves so fast and uses throwaway one-time approaches only, so once you've filtered for a pattern, the next campaign looks completely different.

So, yeah, sorry about the long response, but I don't think Spam has been solved, at all, at least not if you're using an e-mail that's been around for longer than a few years (and appeared at least in one leak).

I am using a small German email provider and I can't remember the last time I got spam.

What's even crazier is that they are so confident in their spam filter that they simply reject mails classified as spam. But that hasn't been a problem since the few years I have been using them as well.

I have a wildcard email on my domain. In fact multiple domains. Multiple of these have emails listed in bug trackers, git repos and in plaintext on my website.

The only filter I have is a couple of regular expressions to block the most lazy of spam at submission time. Even without that I got less than 50 spam mails a day, usually quite a bit less - and almost all of those are automatically sorted into my spam folder. How long does it take you to glance over a low two digit number of emails each day? Not a lot for me.

I think the "spam" problem is overrated. Do you get more spam than me or do you just have different expectations?

Impersonation is “””solved””” by SPF and DKIM, not spam.

Most spam I receive (and that continues to somehow bypass my spam filters) originates from Gmail. So no, spam is not solved, not at all.

Spam is only “solved” on big providers because they mostly accept mails only from other big providers.

---

I would argue that “running an independent mail server” and mass-mailing are two entirely separate concerns.

> Most spam I receive [...] originates from Gmail

> Spam is only “solved” on big providers because they mostly accept mails only from other big providers.

I would say that only accepting emails form other big providers isn't a good solution then.

Of course I can't prove this and is just from my subjectiviy and limited experience, yet...

I've reasons to believe that Gmail users get more spam emails as they report these.

It would be nice if such outsourcing went into other parts of the net (e.g. captcha).

Not only are the blocking. They also send bounce spam although they know with SPF record, that the original mail was send by a server that's not allowed in that domain.

Spam is one of the most difficult content classification challenges for the past 20+ years.

There are approaches that work well, on a small scale, but "solved" is a step too far.

Having been a part-time postmaster for more than a decade by now, I fully agree, and would even go further: Ingress spam is pretty much a solved problem if you play your cards right. ChatGPT et al. might change that again - but the mechanisms you can deploy today are very effective against the current UBE landscape.

The _real_ problem is reliably getting your 100% legit mail into your consenting recipients' inboxes.

> The _real_ problem is reliably getting your 100% legit mail into your consenting recipients' inboxes

It's amazing that having someone in your address book isn't enough in many cases. Like, why?

Because the from address can be forged, probably.

Not really a problem in the age of DKIM, _if_ you want to solve it.

Even with DKIM, all you need is the recipient of one email from one user on one domain (I have hundreds of domains) of your mail server to file a spam report, and WHAM you are blacklisted. So yes, it is a problem even with DKIM. If you have a solution, I would LOVE to hear about it.

If you’re blacklisted, the mail usually doesn’t even reach the spam folder.

Correct.

One of my clients is a construction firm specializing with churches. It is not infrequent for them to be communicating about a project with a church - often to a role account (e.g. info@church.org), which is step 1 towards being filed as spam - where the role account is shared with a dozen or more people. The building manager will check the email on Mon, Wed and Fri and every other day there will be a number of well-intentioned volunteers at least one of which will click on the spam button dooming my client's email into the abyss. Weeks later we find the church has gone with a different contractor, as they "could never get a response" from my client.

We will go so far as to donate a new IT/Cloud system to the church (small $$ compared with the project) just to ensure reliable communication. But then they think we are just trying to buy them.

Now my client has a blacklisted domain/mail server IP, and bids they send out are rejected as spam by the providers to other churches.

Makes no difference if we are hosting the mail server or if it is a 3rd party mail service. As soon as a customer is sold, we try and move them off email into a web-application framework for ongoing legitimate communication. Again a lot of resistance.

Does this happen even if you move your client to Google Workspace/Office 365? Using one of those two should eliminate the problem. Was that your experience?

Life is so easy if you just hand over your personal agency to monopolies and billionaires who don't actually care about you /s

If the alternative is that your construction firm fails and you and everyone else are out of a job - well your email hosting choice is a weird hill to die on.

(the '/s' means 'sarcasm' and that wasn't the user you replied to)

Doh missed that, reading is hard!

No. Workspace gave no apparent protection. Why/how would it?

Presumably most of the firms clients are themselves hosting email on Workspace or 365. And Google/MS might treat email originating from their own systems differently than inbound email originating elsewhere.

Also I'd expect all the firms competitors to be using Workspace or 365, so if using them gives no apparent protection, presumably they should be suffering from this as well?

I file this under “you can’t have a technical solution to a social problem”. We can do all we want to protect e-mail, but when it comes down to it, someone is going to figure out a way around it and ruin it for others.

The current situation is that we have technical solutions for authenticating smtp sending domains. But there will always be someone who flags an email too quickly or just wants to spite you. Or someone that will send spam regardless (or hack an account, etc...). And so we’re back at square one.

This is especially true for situations (like email) that aren't subject to market forces. Because sending email is so cheap, the return rates can be very small to still justify sending spam. There just isn't any market pressure to keep it contained...

Hashcash was originally proposed to add some form of cost to sending email. Something similar could be a great way to get mail from legitimate people through. Spam wouldn't scale but the average person would only pay in a bit of CPU time/cost.

“spam wouldn’t scale” - unconvinced on this: spammers already mostly use other people’s compromised machines to do the sending; there is no cost to them here.

If you charge some cost per mail (whether that's CPU time or actual money), users/teams would check their spend and optimize accordingly. They'd notice runaway spend and act on it. The only reason why mail servers become compromised and nobody notices is that bandwidth for mail is generally too cheap to meter. On any 10s of megabits connection, sending a deluge of spam is trivial.

Not even that, you can become the victim of a "noisy" neighbor if someone on the same IPv4 /24 sends too much spam (some of the common blacklist providers will do entire netblocks if they get enough complaints)

Yeah, it may be a combination of “not everybody uses DKIM” and “too few users actually use their address book”.

from can be forged, but spf is there so only valid servers can send mail (or at least non valid can be filtered).

SPF can break with automatic email forwarding (though DKIM usually shouldn’t).

But to exploit this, an attacker would need to have the address book of every account they are trying to spam.

That's relatively trivial. Send N^2 emails, pay attention to the bounce backs.

There's almost no cost to send an email except time so even when n is large, this does not prove nearly as intractable as one wants it to be.

I’d guess the main issue is, the big guys don’t care about you. Like, at all.

If they pass their log through a scanner, come up with a few dozen small servers that look fishy (e.g your IP is contiguous to some other spammer’s IP), and yours is rope in as false positive, you’re banned.

And from there you’ll have to convince Google that you’ve done nothing wrong.

> And from there you’ll have to convince Google that you’ve done nothing wrong.

Imagine the conversation with their support when you ask why bob@domainx.com's email was put in spam folder of jan@domainx.com's email, as both are hosted by Gsuite and presumably entirely within the Google network.

The answer, after two levels of support, was, "We don't understand why, but we can open a ticket with the developers." In the business world, you don't have time to wait for that, especially when non-technical business users are making mistakes or losing business because they can't function properly.

I've gotten emails from old friend accounts that were totally taken over for spam purposes. Usually a low tech friend who just creates a new account.

I moved a Gsuite client to Zoho because once or twice a week, an email from one person at the custom domain to another person in the same custom domain (company) would have their direct email get put into spam in Gmail.

You would really think that Google would whitelist emails within the same custom domain which is paying for Gsuite service. Maybe that has changed, but it was definitely a problem 5 years ago.

And we're not talking about an email which had some copy/paste of spam, we're talking about a one or two line sentence giving an instruction or asking a question to a colleague.

https://support.google.com/a/answer/2368132

> Bypass spam filters for internal senders

Is this what you are looking for?

Shocked that an article is required to explain what should be solved in the product.

It is solved by the product. You tick a box, and internal mail is bypassed.

The issue here is that developers and IT folk seem to think that email is easy because the know IT. Email is a complex, and old protocol that has many nuances and, believe it or not, phishers are smart; end users are naive, and often reckless. Spammers, be they benign or otherwise, can be incredibly lazy too, but don't think for one second that you, as a postmaster, are cleverer than they are. Spam and phishing are getting harder to beat. Businesses like Proofpoint, Cisco and Mimecast, along with the likes of Google and Microsoft, are investing heavily in trying to beat these guys, but the reality is that they are always at least two steps ahead.

Check box or not, my point is this should be default behaviour, I.e., don't mark same domain as spam, and instead make that an option to enable.

Drives me nuts having to leave a UI to google (ironically) the issue to find a doc to enable a check box.

At scale, that’s not an option. If you have external services that send mail to your Workspace tenant, there is a risk of compromise to the sending service, especially if it is outside of your control. The sending service could have SPF records that permit sending mail on your behalf. That mail needs to be scanned. There is also the threat from bad actors inside your perimeter. Sure if you’re a small operation, or using Workspace as a personal mail host, this may seem like overkill, but I can assure you that the majority of orgs that use Workspace (it a business too after all) would prefer the status quo.

Don’t know about GP but this is what I was looking for. Thanks!

Google's own emails about expense reimbursements were systematically flagged as SPAM on gmail and they knew it.

They still didn't care at all!

That would demand maintaining an address book. Most mails I receive are from services which I have never written to, and I don't see a reason to extra maintain their data when I already have their mails.

For sure. I’m contemplating a scenario I’ve had multiple times “No I didn’t get it” “Okay let me try again, did you check spam?” “Yeah” “Maybe add me to your address book” “Nope still nothing”

Well, ChatGPT could write a mail server too, so what are we even trying to achieve any more?

Just let ChatGPT send emails to itself, using mail server and client software that it wrote itself, and the rest of us don't have any need to handle email ever again.

I believe what the parent meant was that SPAM blocking relies heavily on keywords, and strings. With the advent of chat gpt we go beyond a/b testing and push it to into the every infinite way of someone conning you to click on something through email.

I've been doing it for well over 20 years now.

Ingress - out of the box rspamd is pretty decent and it is rather configurable.

Egress - DNS (A,AAAA, PTR), (E)HELO, SPF, DKIM, DMARC. "IP Reputation".

There are absolutely no shortcuts and yet most of the problems I diagnose regarding email delivery will find a missing PTR record or a miss-configured (or non configured) HELO. You cannot be lazy when it comes to email. SPF + A + AAAA + PTR and (E)HELO is a minimum.

There is one thing that you cannot generally, personally mitigate and that is being on a deny list due to your IP address. This one is a bit more tricky to deal with. You might have to use a relay. Another mitigation might involve IPv6.

> Egress - DNS (A,AAAA, PTR), (E)HELO, SPF, DKIM, DMARC. "IP Reputation".

Any insight into why this is a giant ball of complexity? I’ve had to setup spf, dkim, dmarc a couple times now and man… design by committee?

There is no design committee, they just keep adding stuff over and tell us is definitely the one that will fix security/phising/privacy/spam issues SMTP suffers.

I think it’s a miracle E-mail still works at all this far into web whatever-point-oh we’re in.

That is not that complex...

They solve different things. Email is very old so obviously things are going to be tack-on rather than baked-in.

The standards are fine in terms of achieving their goal, the hard part is getting the ecosystem to follow and make everyone's life easier.

Sorry for naivety, what is HELO in this context? I've been running a mail server with the other aforementioned things set correctly and tested; I've been able to reliably deliver mail to all of the big senders for years now, albeit on a small scale.

Am I missing something not knowing what HELO is ??

HELO (or more usually EHLO) is the initial greeting of SMTP protocol which includes the mail server's address.

AFAIK plenty ofservers will outright reject all mail if you only have an IP address and not a domain name there.

As a user, I'd rather my receiving email provider (eg. Gmail, iCloud, Outlook) be less restrictive in their filters if it means individuals can host their own mail servers and we can have a more egalitarian internet.

Extremely filtered emails is practically the same as a social network determining what I can see at this point.

Only it’s not a social network, not even close.

Web standards have advanced to a level that walled gardens are replacing email. Email itself is still stuck on IE6 level standards. And SPF, DMARC and DKIM are a confusing mess to deal with.

Email is showing its age, but are SPF, DKIM, DMARC really confusing?

I've had a company tell me the other day that you can't do DMARC without using their services. It's just too complex. Why is that - we don't run our own mail servers and we use Google mail for our organization. I'd think Google would do that?

If a company tells you that it must be true. /s

Sounds like a salesmen who doesn't understand DMARC himself was trying to sell it by labeling it too complex.

From a purely technological point of view DMARC isn't that complicated. It just specifies how to treat DKIM and SPF results (with a bit of rather simple configuration). SPF is basically a list of ip's you own published on your domain (if the email was sent by that ip, you vouch that it was sent by you) and DKIM signs the email with a private key and you publish your public key on your domain so everyone can verify that this email was signed by the domain owner. SPF might fail if your mail gets proxied (as now its a different sender ip that you didn't vouch for) and DKIM might fail if the mail got modified including headers (because the signature can only be verified for exactly the original headers+content). So if you're sending email for someone else it gets a bit tricky, but for your own emails it's certainly not "just too complex" and boils down to a few line long configuration file, a list of ip's and a private/public key pair for signing emails.

Yeah, I need to do more research on it, thanks for your advice. It seems like this must surely be possible to set up on our corporate Gmail admin account.

Not my experience. I have run a personal mail server for about 15 years. My mail has almost always been delivered just fine. The network my mail server IP has been in a blocklist less than handful times, but this has always been temporary and resolved without my intervention (perhaps my hosting provider is on top of this).

On the other hand, I'm also sending email for a client through AWS SES, and those emails are regularly blocked due to their IPs being on a blocklist. And indeed I also get spam delivered through AWS SES. I understand that it is a cat & mouse game for mail service providers to prevent spam being sent through their services. My point: switching to cloud providers doesn't magically solve your mail problems.

Aonther issue with large mail account providers (gmail at least), is that they will accept & deliver your message, but just "deliver" it to the spam folder. It's so annoying if mail is accepted but you don't know if it is actually seen by the recipient. This erodes trust in email. I understand mail servers don't want to tell spammers whether their email is accepted/rejected, but I would prefer a few more spams in my inbox if I at least won't miss emails.

>It's so annoying if mail is accepted but you don't know if it is actually seen by the recipient.

that's what those embedded 1x1px images are for

Some users have embedded images disabled in email client by default.

some users have javascript disabled, some have email clients disabled. it's not foolproof but works for the majority of users, i.e. >99%

Any any well designed website will work with javascript disabled unless it is absolutely needed for the task (i.e. we are talking about an actual app).

There is also no guarantee that the image being loaded means that the user opened the mail and it wasn't just preemptively cached by the server/client. There is also no guarantee that the user opening the mail means they read it.

have you seen the web lately? javascript is increasingly being used on backend. not just for apps, for static one page sites too. those who disable javascript do not have a pleasant browsing experience and are forced to enable it often

I'm not here to guarantee anything, just mentioning the technology exists and is still widely employed. there are no guarantees with messaging app ticks either

Are you sure those are GDPR-compliant? Just because you can do something doesn't mean that you should.

I have no idea, but doubt they are mentioned specifically

I am always reading and never experiencing this (knockonwood).

I'm self hosting email servers for multiple domains for now nearly 25 years. When moving to a new hoster with new IP addresses (especially Hetzner, as it is cheaper and seems to attract more malicious people), I have to contact Google, Microsoft, Yahoo and one or two other big providers to clear my IP address range. This has to be done only once and usually is resolved within hours (faster for automated sites). The bigger ones have a web page to do this, others can be reached via email and are always replying quite quickly. So after a day, everything works and once it works, this never changes and mail gets accepted.

These servers are being used by multiple users and businesses and none have reported any problems. For decades.

I am only using dedicated servers though and using everything in the book: DNSSEC, DMARC, SPF, DKIM, proper DNS records, TLS with valid certs and a correctly behaving SMTP server...

How are you contacting Google, Microsoft and Yahoo regarding that?

They provide webpages (forms) where you can insert your info and they'll unblock your ip...

No, that kind of software optimizes a very important problem. It’s quite cumbersome to set up all components of a mail server by yourself. At some point you start hosting a domain for a friend. Then then friend wants to create some mailboxes, forwardings and so on by themselves. So you just give them SSH and tell them to edit the postfix config? Having a web interface that does it all and doesn’t break things is very important.

Who does this...?

Giving out free email addresses to friends and family? Me.

Hey, we are starting this charity and need email for 15 people, what should we do? - order domain, create admin-account in the web interface, pass it on, done.

This problem is solvable without a web interface: https://manpages.ubuntu.com/manpages/bionic/man5/dot-qmail.5...

How many people understand that man page, and how many people understand a web interface? I think it’s a clear winner.

I hate to be an elitist jerk, but if you can't understand a man page describing a single file format then maybe running an email service is something best outsourced.

It is usually not the same people who run it and who create users. You don’t want to bother the admin every time you add a new email address.

But an admin UI is more than just for creating mailboxes. It can also manage quotas, the spam filter, provide setup instructions for the end users, …

I hate to be an elitist jerk, but how am I going to teach my mum to get her email using THAT?!

simple-nixos-mailserver... One step deploy. Hasn't broken in years (it's never broken for me). Extremely stable. Can handle all your use cases. Declarative config, so no messy state.

I hosted my own mail for some time and got into trouble with the school of my kids because they did not receive my reply to their mail.

I self hosted because I wanted to prefer not to be part of the huge o365/Gmail/iCloud monocultures.

Last year I moved my mail to an old fashioned shared webhosting account at Hetzner. Very happy with it!

> Last year I moved my mail to an old fashioned shared webhosting account at Hetzner. Very happy with it!

How exactly is that solving the problem? If anyone does something remotely spammy from that ip, your mails are spam again too. And you probably got lucky that the ip you're sitting on was warm and trusted to begin with. You didn't really find a solution, the problem simply hasn't occurred yet for you or you are not aware of it yet.

GP is right. Self hosting email sending (and by that I mean any solution where you control the mail server) doesn't work unless you accept that you will randomly end up in spam folders and sometimes not delivered at all.

From my experience shared web hosting accounts have the worst trust. You get the IP range of a public cloud provider, but you also share it with random people, that will send spam at some point.

So either get a package at some of those smaller providers with a dedicated IP, or go to one of the big providers (Google, Microsoft) and use their highly trusted IPs, that everybody whitelists by default.

I think you misunderstand. You cannot have a mail service on shared webhosting. It’s like 1999: Upload PHP using FTP, the end.

Instead, the hosting provider will operate a mail service for you, much like Microsoft 365, G Suite and the like. They will also take care of the IP reputation, SPF, DKIM and all.

Shared hosting does not typically run an external mail service. It's just Exim or whatever running on the same box as the shared Apache.

Maybe they do, but I would be surprised seing managed email sending in Hetzner's portfolio as part of shared web hosting. It's just not what that company typically has to offer, that's why I expected they just provide you the infrastructure, but ip reputation and so on is your problem. Taking care of IP reputation and successful delivery is a massive undertaking, especially if you let customers send whatever they want.

However, I dunno why people only mention the cloud providers, while services like MailChimp, Sendgrid and others offer this for mass emails and companies like fastmail for personal users. The cloud isn't really necessary at all.

The business for the web hoster is to provide me with some MBs for a website, plus a database, and also to provide me with some mailboxes on my domain.

Many small busineses have their 'online' presence like that. If I have mail deliverability issues, it's not my problem any more, I can call Hetzner. And before I started using it I was sceptical because if Hetzner did not fix the reputibility of their IPs or the spam from their other customers, it would be a problem for me. But so far, it worked quite well!

> These days I'm very happy to pay somebody else to run email services using my provided domains.

I was in a similar boat, though I bailed much longer ago than you. I think "Big Email" is happy to let enough spam flourish in the world so that it presents enough of an excuse to tighten the screws to make it effectively impossible to run your own server... just so that more and more mail goes through their servers to be mined for personal data. That's why I don't use Gmail any more, but it wouldn't shock me to find out that Apple is doing something with my email data too.

Counter point... Make email hosting as easy and abuse free that many more people can host their servers( for themselves or for community) breaking the big tech monopoly.

Some big mail carriers who are more FOSS-friendly should create a service where they will deliver your private mail-server mail to Google and Microsoft inboxes for you. So you can use your own mail server for every other mailbox with more reasonable filtering protocols, and when you have to send mail to a Microsoft inbox you just send it to them and they’ll forward it.

They could also provide Cloudflare-like spam and DDoS protection for receiving mail

You would still deal with the deliverability challenges of getting your private email server trusted by everyone. It will work 95% but there are a bunch of weird edge cases due to old IP blacklists, bad spam rules, etc that will cause you problems and basically require a greybeard email expert to diagnose and fix. Sadly, I think it's just too hard to run your own email server these days.

Why are the email providers so bad about this? When I check my "spam" filder in Gmail, around half the email is not spam. Is it because humans are so bad at classifying spam?

May I ask, why are you still using Gmail then?

This is an honest question, in my opinion the only way to make Google more friendly with self-hosted mail servers is their users complaining about or leaving Gmail.

I can't even think of another provider. And Gmail is otherwise quite amazing.

google has one of the worst spam filters. It can be done much better, and google could do it much better. But I suppose they are just big enough now. Can't reach your contacts that use google? Well, just sign up to google as well, then you can reach them. People like you are the reason google can behave like that.

We use Mailgun as outbound relay for self-hosted email stack, and no more problem with delivery. As a home user you can even stay within the free tier and never pay a dollar for it.

I refer to this as ‘the mail cartel problem’.

Apropos

Reply on news.ycombinator.com