Hacker News Reader

I just recently started self hosting email. I settled on relaying through gmail to all hosts except a whitelist of hosts I know won't blackhole me - mostly just a few niche domains of friends who do the same thing or small organizations they're a part of. Have you considered this? It's not optimal but it's a practical compromise. You get to keep your From: field as your domain too if you set it up right. I couldn't make it into Microsoft inboxes at all before I set up the relay but now I can.

- Make a gmail account just for your email server, which forwards anything incoming to your host in case someone emails it directly (you'll be able to discover this gmail address if you dig through delivered emails' headers but it won't be in the From: field)

- Let Gmail authenticate with your SMTP server in Gmail's advanced options, and make sure the options are checked to retain the From: headers of relayed emails

- Generate an App Password and set up your mail server software to use Google's SMTP relay with the email and app password

There are other services to do this too I'm sure, but I'm happy with Gmail for now. And you can always transparently switch it out to another service if Google pulls something.

I plan to write a blog post on the whole process of setting up a mail server to configuring it with a gmail relay like this.

I did consider that as an option, and I tried routing email through a number of other delivery services, but ultimately I rejected it for a few reasons:

I was hosting email not just for myself but for a few other people, and that gets tricky to route through Gmail;

Gmail could change their policies at any time and give me a really bad day, potentially when I can't respond to it in a timely manner;

If routing email in this way ever triggers Google's abuse mechanisms, then potentially I'm losing access to a lot of the Google network, and while I don't use it for anything personally, sometimes I have to work with companies that do;

After spending tons of hours dealing with Gmail-related headaches despite not using Gmail myself, relying on them to get mail routed felt like a deal with the devil.

I've instead helped a bunch of people get set up with Fastmail. I love Fastmail, they're great, I miss hosting my own email but they're the next best thing. Fastmail must be handling enough traffic that they're hovering above Gmail's piss-off threshold, and really my experience with them has been extraordinarily good. Everyone I've set up over there has been happy with them too, save for one person who got told by the next IT hat-wearer that "everyone's using Google Workspace and you should be too" (and then immediately ran into a problem during setup that snowballed into a big hairy mess).

>felt like a deal with the devil

Why not if you send mail to hell? Who else do you expect to deal with?

Exactly this. I enjoyed the technical aspects of running my own mail server. I hated having to constantly manage black lists and dealing with "postmasters" to get my email to be consistently delivered. Some spammer manages to hack a server on the same IP block as you equals trouble for months. I just couldn't justify the time spent and the non-delivery any longer.

Even google is aggressive about IP bans on other services.

There was a time my home IP on Comcast was on some blacklist with good. Every video was run through captcha. Searches too.

Comcast was useless to give me an IP in a new block.

What solved it? I signed up for gsuite free at the time and moved my email (from a colo) from “on prem” to them. Suddenly my home IP that’s used to access their services is cherry and no longer suspicious.

Note: that doesn’t seem to work anymore though. I regularly proxy some traffic through a linode and google does the same thing. Everything behind a captcha that’s stupid difficult to clearly 60% of the time. I hate traffic lights.

I doggedly still run my 20+ year email platform for self-use. I ran into the same problems with Google/Gmail. I am ashamed to say that the solution I settled on was to just give up and pay them for smarthosting. I still host my own servers, but my exit pipe is through gmail. My life got easier.

The rule is pretty much every interaction you have in the Googleverse is easier if you just pay them something. It's pay to play internet, and yeah, it's a problem.

First of a

> It's pay to play internet, and yeah, it's a problem.

1. How much do they charge?

I'm genuinely curious. I don't self host, but use a 3rd party (fastmail). I send very few emails to people I don't know, so personally, I don't run into issues with having my email sent to spam.

2. I don't think paying in itself is the real problem. I think it's more a matter of who you pay and why you pay.

- You have to pay to register a domain name. - You have to pay to host your own server (whether your using a hosting service or hosting from your basement) - You have to pay to have gmail not mark your email as spam - ok, I'll admit, this is a little silly, but you also have to pay (via a stamp) to have USPS send letters to their recipient

3. Perhaps because so many people use and trust (whether they should or not is another question) gmail, it makes sense to pay in some scenarios? But obviously, for personal mail servers, I agree, asking to pay to play is a bit of a stretch.

Can you explain why? I looked this up and still am clueless as to why.

> Gmail really pioneered the "nah, screw you" approach to this.

From my POV, Yahoo pioneered this way before Gmail. 10 years ago when I was managing outgoing mail infra for a company that probably sent 100k emails per day (mostly purchase receipts and subscription notifications), I was intimately aware of deliverability characteristics for Gmail and Yahoo, which were our two biggest destinations by a large margin (if I recall, Gmail was about 60% and Yahoo was about 30%). Gmail, despite being the overwhelming majority of our traffic, was hardly any problem at all, whereas Yahoo would regularly give us the dreaded 451 response that indicated that your mail was going to be held for 2-24 hours and then randomly rejected or accepted.

BTW, if I were to run my own mail infrastructure again, I'd definitely use someone like Sendgrid with a dedicated outgoing IP with a clean reputation. It makes a very big difference. We eventually ended up with our infrastructure split across multiple IPs by function (e.g. receipt emails were the only thing ever sent from one specific IP because we wanted to ensure deliverability).

The trick to get reliably mail delivery to Yahoo back in the day was to be spending several thousand dollars a month buying ads through Yahoo.

Then when your mails to Yahoo got repeatedly blocked by Yahoo and you got tired of having to repeatedly report the false positives and and get unblocked only to get blocked again a few weeks later you have the person in charge of your ad spending call up your ad sales rep at Yahoo.

You asked the ad sales rep "Why should we keep spending $X/month on Yahoo ads to try to acquire customers, when Yahoo mail keeps blocking the receipt emails, setup instruction emails, and response emails to support questions we try to send to those customers which angers those customers and they cancel (or even try chargebacks)???".

What happened then is the sales rep puts your ad manager on hold for a few moments, then comes back with someone high up in Yahoo IT management conferenced in, and explains the situation. The high IT manager puts you all on hold, and comes back in a few minutes with someone who actually deals with maintaining the spam filters conferenced in, explains the situation, and tells the maintainer to add your domain to a special whitelist of mail that is always accepted.

I suppose that the only real problem is sending, not receiving, nor setting up software.

If I were to set up my own mail server, that would send outbound email through Sendgrid, AWS SES, or maybe some other established and trusted sender.

Building trusts with behemoths like GMail or outlook.com is too involved an affair to be worth it in the general case, and not even through some malice on the side of the latter; it's just the reality of a medium not protected from spam.

I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one. FWIW, I used to get a fair bit of inbound spam from Sendgrid IPs, so they floated in and out of my greylist on the regular and I wasn't too surprised when they didn't turn out to be a good option for my outbounds. I was surprised about Mailgun; I don't recall receiving any bad traffic from them ever, and the couple of times I've used them, they seem to focus a lot of energy on not being a source of trouble.

> I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one.

I mean, you can't reliably send email to gmail from gmail. It's not really the origin that's the issue, it is that gmail spam classification is quite bad. Good emails from known recipients you've corresponded actively for years will continue going to spam in gmail even as you mark them not-spam for the thousandth time.

So yes, sending to anyone @gmail.com has decents odds of ending up in spam for no reason. That's just how gmail is.

But no, sending it from your own email server doesn't make things worse. I run all my own email infrastructure and sending to gmail addresses works just as well from my server as it does from gmail itself.

> Building trusts with behemoths like GMail or outlook.com is too involved

There's nothing like that actually. I have e-mail server running since 2000s. When gmail appeared, everything continued to work as expected and it did for many years. Around 2018 or 2019 I noticed that my e-mails started going to spam folder. I didn't send any bulk e-mails, I was not on any RBLs, I was not compromised (I actually was very careful to block outbound SMTP from any user except the one running MTA), and of course had SPF & DKIM set up.

It looks like their spam filtering is just arbitrary and feels like done on purpose to discourage running personal mail servers (and looking at comments it works really well).

Honestly, I think some kind of campaign is needed to put them in place, like starting blocking e-mails from gmail (maybe responding with a message encouraging to switch account). I remember in the past steps like that were done, but feels like today people are more acceptable of centralization.

They certainly are discouraging from running small mail servers, because it's harder to hold a small installation accountable. A spammer is usually a tiny installation which would disappear overnight, being run on random botnet nodes.

I don't think they explicitly are trying to quash smaller independent mail servers (personal or commercial), they just see what spam statistics show, and update blocking rules accordingly, maybe fully automatically, using ML. The fact that they also quash small-time independent competition is just a nice (for them) side effect.

They operate since 2007, they can track who is reliable and who is not. It's not like IPs change[1].

[1] yeah, they might be some people trying mail servers on dynamic IP, but I'm not talking about those and Google would have some justification not categorizing them as reliable. I'm talking about using business level ISP offering with a static IP address and matching RevDNS.

In many cases I'm not sure it is even possible with Outlook.com.

Sending mail from my domain to my Outlook.com address it kept going into spam seemingly no matter how many times I did any of these things:

• Find the mail in my spam folder and mark it as not spam.

• Whitelist the sender.

• Reply to the mail.

If google can't automate it then it doesn't exist for the most part. They bend over backwards not to deal with actual humans, and they're so big now they could not care less if they lose -1 human or business, it simply doesn't matter unless you are rich enough to know someone in the 1st tier executive team.