Hacker News Reader

Comment by kazinator | Hacker News Reader

You can't run your own e-mail, or not entirely. It's practically impossible to send SMTP from your own IP address. For sending SMTP, you need to go through a smarthost that has reputation.

If your ISP provides you with an e-mail setup that you can use with a conventional mail client where you enter IMAP4 and SMTP credentials, chances are you can use that for SMTP sending. I.e. from the perspective of sending mail, your ISP can't tell that you're a server; it thinks it's just Outlook or Thunderbird connecting to it.

Receiving mail is no problem; your ISP just must not be blocking port 25.

It's handy to give yourself mobile access. When I send mail from my phone, it connects to port 537 of my own mail server which provides authenticated SMTP over TLS. It forwards to the aforementioned ISP. (I can't connect directly to my home ISP's SMTP server from my phone because the phone is on a mobile network unrelated to that ISP; the ISP's SMTP forwarding servers are firewalled so only the subscriber addresses can talk to them.)

View on HN

Mox's FAQ addresses this question:

https://www.xmox.nl/faq/#hdr-won-t-the-big-email-providers-b...

Won't the big email providers block my email?

It is a common misconception that it is impossible to run your own email server nowadays. The claim is that the handful big email providers will simply block your email. However, you can run your own email server just fine, and your email will be accepted, provided you are doing it right.

If your email is rejected, it is often because your IP address has a bad email sending reputation. Email servers often use IP blocklists to reject email networks with a bad email sending reputation. These blocklists often work at the level of whole network ranges. So if you try to run an email server from a hosting provider with a bad reputation (which happens if they don't monitor their network or don't act on abuse/spam reports), your IP too will have a bad reputation and other mail servers (both large and small) may reject messages coming from you. During the quickstart, mox checks if your IPs are on a few often-used blocklists. It's typically not a good idea to host an email server on the cheapest or largest cloud providers: They often don't spend the resources necessary for a good reputation, or they simply block all outgoing SMTP traffic. It's better to look for a technically-focused local provider. They too may initially block outgoing SMTP connections on new machines to prevent spam from their networks. But they will either automatically open up outgoing SMTP traffic after a cool down period (e.g. 24 hours), or after you've contacted their support.

After you get past the IP blocklist checks, email servers use many more signals to determine if your email message could be spam and should be rejected. Mox helps you set up a system that doesn't trigger most of the technical signals (e.g. with SPF/DKIM/DMARC). But there are more signals, for example: Sending to a mail server or address for the first time. Sending from a newly registered domain (especially if you're sending automated messages, and if you send more messages after previous messages were rejected), domains that existed for a few weeks to a month are treated more friendly. Sending messages with content that resembles known spam messages.

Should your email be rejected, you will typically get an error message during the SMTP transaction that explains why. In the case of big email providers the error message often has instructions on how to prove to them you are a legitimate sender.

That FAQ doesn't address anything. Suppose you're in a blacklisted block. Now what? Most residential IPs are blacklisted.

When I say I'm self-hosting, I mean I have a machine under a table right here in my home: True Scotsman's cotsman's self-hosting.

You can still do that by tunneling the IP address of a cheap VPS to your home network

When it is blacklisted, the user/owner of the IP must go to each "anti spam" provider that blacklisted the IP. I.e. spamhaus, they have a page[1] to check if an IP is blacklisted as well as asking to remove the IP from blacklist.

[1] https://check.spamhaus.org/

Some blacklists say "this is a residential subscriber IP". Some mail servers block based on that. An IP cannot be removed from such a database.

https://www.rbl-dns.com/dul

I drop SMTP connections from servers that simply do not have matching forward and reverse DNS. This rule eliminates like 90% of spam. It's a good rule and I won't make any exceptions. There's no way to contact me. Your bounce message tells you what you have to do: get your DNS ducks in a row.

... and that's nearly impossible if you're on a residential connection and hence have no control over your reverse DNS... And who wants their mail server to self-identify as d203-0-113-5.res.fubar.isp.net ?

That's assuming your residential ISP even bothers to assign a generic PTR record to your IP.

Yep, this did not work. I tried for several years. The only thing that worked was manually filling out the required forms. Not sure if it changed lately. Out of the self host mail business for some years.

> the user/owner of the IP must go to each "anti spam" provider that blacklisted the IP.

Even that doesn't work all the time. hotmail is currently bouncing emails from me[0] even though Microsoft's own sender reputation thing[1] says my IP is in good standing.

[0] with a link to [1] just to rub it in.

What finally forced me to switch to a 3rd party for SMTP (outgoing) was a blocklist (UCEPROTECT I think) that required you to pay to be removed and my mother-in-law's email provider (AT&T) used it. My wife couldn't email her mom which was a no-go.

Blacklisted residential IPs add 0.1 in the default spam assassin config

Spam from residential IP's should never reach SpamAssassin. The mail server should be rejecting the SMTP connection. SpamAssassin is something which deals with mail that has been accepted by a server: i.e. delivered. It shouldn't need to have any rules about residential IPs; what's the point.

Residential IPs are spammy, so if for some reason you've decided you're going to let SpamAssassin to handle them post-delivery, it would make sense to give them a high score.

> Spam from residential IP's should never reach SpamAssassin.

If the residential IP is in the MX record for the domain, even more so if the domain passes DKIM, why not?

That's an interesting heuristic. Given a host connecting from an apparent dynamic IP without matching forward and reverse DNS, we could take their purported e-mail domain (from where? SMTP hello? Or domain part of MAIL from?) and fetch the MX record to see if it points to that same IP address and use that as a whitelist criterion against being dropped as a residential IP. (On the hypothesis that they are trying to run an earnest self-hosted mail setup.)

However, if the host passes this check, and all other tests such that we decide to accept the mail for delivery (to be further processed by SpamAssassin), at that point why would we want to apply any score in SpamAssassin regarding the residential IP. We already decided to pass it.

This FAQ is in complete disregard of reality. Almost all IP ranges of server providers are blocked. Getting a clean IP is close to impossible.

Big providers often only support their own forms and ignore open sources trust providers.

Small providers often do not maintain their email services which will simply auto spam your mail/domain, when it does not come from the big 10 providers.

I did not find this to be the case.

I set up a mail server with NixOS 5 times in a row with 5 different Hetzner Cloud IPs and each of them arrived fine at Google.

With fine you mean not in the SPAM folder the first time? That would be an improvement.

Also works for Microsoft services?

Yes, "fine" means not in Spam.

> Also works for Microsoft services?

No, Microsoft seems to have an allow-list for servers, and ban everything by default (the laziest of approaches).

But that's OK: Because Microsoft does it so poorly, Microsoft email users check their Spam folder all the time anyway.

Source of your claim? I'm monitoring blocklists of about 20 different VPS providers, most of them are completely clean, some are in one policy blocklist because they don't allow outgoing emails and only couple are on 2+ blacklists (of 67 monitored) because there's some noisy neighbor on the /24 subnet.

Sadly only my personal experience across multiple years. I think I was selfhosting my mail for about 5 years. Multiple Hetzner root servers using mailinabox and later mailcow. All self checks green.

Bizarre claims. I've been running my own server for the last 25 years or so. Only once when changing server IPs I've encountered an IP that was blacklisted on some lists, and even then it only took a day or two to remove it.

Maybe it is (was?) a German problem. Here are some providers I know which Autoblock custom servers:

* web.de

* gmx.net

I also have to say that I always used a hetzner root server. Moved multiple times due to an upgrade.

I ALWAYS had to manually apply for removal of my Webserver. It worked for Yahoo. At that time it did not work for Gmail and Microsoft. I no longer was blocked but if I was writing for the first time to a recipient, I landed in the spam folder.

The software I used mailinabox and mailcow. Both had self checks. All green. I also used external scanners to check my config, all fine. You can check my GitHub (razemio). I even contributed to some issues for mailinabox.

This is not only true for selfhosting but also small providers. As an example:

* mailbox.org (auto spam Gmail and microsoft 2018)

All of this was a long time ago. Maybe I am just depressed from the bad experience and the FAQs are telling the truth. However it is hard to believe for me.

I am selfhosting since 1997 and I am working in programming / DevOps.

Sure, it's not technically "impossible", but like...

> It is a common misconception that it is impossible to run your own email server

... the FAQ then goes on to give all the reasons that argue it's really really hard and probably not worth it for most people.

Don't they say that those perceived problems aren't actual ones?

Using an ISP's SMTP is an incredibly obsolete and problematic concept. Poorly authenticated with even worse deliverability. It was a bad idea even 10 years ago and it's just horrid right now.

Use your email provider's SMTP, even if it's you yourself.

This just isn't true, of course you can, you just need to use a hosting provider or ISP that allows it. Plenty do.

It's not whether the hosting provider or ISP allows it, it's whether the address they give you has reputation so that mail servers all over the world allow connections from it.

tl;dr: If you set up an email server. You just send a few emails to gmail, etc. and you'll know if they are accepted or not. If yes you are set, if not you investigate the problem, maybe really try another IP and eventually fix the issue. It's not like all of a sudden your email becomes undeliverable, unless you start sending spam. Imagine what it would mean for all the people using the email address of a small ISP, some university, etc.

Sorry, but that's FUDish. The reality is if you do a proper email setup (DKIM, Reverse IP, etc.) you will be fine.

If you happen to actually get an IP that somewhat recently happened to be an email server, that was also sent out spam, which isn't something that's likely at all then you'll notice very quickly (just send an email to to gmail, etc.) and what you'll do then is tell your hosting provider you'd like another IP address, because it's not fit for your purposes.

I've been running, moving, switching IPs, providers, domains since 2005 and still am and there is just SO MUCH FUD. It's not hard. It's a one time thing. Personally I never ran into IP reputation issues ever. These are email addresses used in a professional capacity (B2B, communication with governments, etc.) as well as private use ones.

Pretty much every ISP, every university, etc. runs their own email server. Many companies do. Many private people do.

I have run them on the side for those 20 years now, partly as a hobby and so far the uptime was higher than Gmail's and since I use them for private, professional and sometimes for government communication I am dogfooding it and I would have very much noticed if anything bounced.

I have gotten bounces when a setup was initially broken, like when I do something like sending a test email to Gmail and that was off.

The reality is that IP and domain reputation aren't really great ways to filter spam anyways. Yes, it adds, but what makes you think that nobody sends emails from Gmail, a university or other stuff? What makes you think that spammers use static domains, etc.

Heck, not even DKIM and SPF are any guarantee. People will spam you from servers with extremely good reputation. Looking at my spam box most of them are from situations where accounts obviously simply haven't been blocked yet.

No serious spam filtering is done with IPs or domains being an "all or nothing" thing.

Also it's a two-way street. If a user of some email provider doesn't get their email and it becomes known people will be wary of it. And nobody expects the email landscape to stay static. There are newsletter and transactional email services all over the place, lots of marketing platforms running their own email servers and so on.

It's not like everyone does something magic, nor does everyone have connections, money or time to talk to all these companies. An email service not accepting emails won't exist for long.

And something that's also important to realize: If you do start using a transactional email service they oftentimes will make you pay EXTRA for a custom IP so you DO NOT share it with others, so you get BETTER reputation than the cheap one. And you configure your own domain with it. So why wouldn't these emails get delivered? And many of those don't run their own data centers and not all of them have their own IP blocks (though some have).

It's just if you couldn't even do regular private emailing, emails would not be the thing every website uses for login and communication.

If you think that you are "set" when your self-hosted setup passes the test case of communicating with Gmail and a few other big providers, you're saying that it doesn't matter if you cannot communicate with smaller providers, including other self-hosted guys like you. If they have any trouble reaching you, why they should just effin' use gmail! That works fine!

> The reality is if you do a proper email setup (DKIM, Reverse IP, etc.) you will be fine.

You're not getting reverse DNS on a dynamic home IP.

> Pretty much every ISP, every university, etc. runs their own email server.

Yes? And what did I say: if your ISP has mail servers for you, it can simplify things greatly if you use them.

> you're saying that it doesn't matter if you cannot communicate with smaller providers

Smaller providers will generally not black hole legitimate message like Hotmail does. They have (paying) customers awaiting those messages. Junk folder? Sure, that can happen sometimes.

> You're not getting reverse DNS on a dynamic home IP.

I don't think anyone here is suggesting running a mail server on a dynamic IP.

> And what did I say: if your ISP has mail servers for you, it can simplify things greatly if you use them.

Only if you want to be using their domain and if you're not sending (too many) automated messages.

Running a mail server dynamic IP is the primary self hosting option for most people. I've been doing it without problems for 15 years now.

> Only if you want to be using their domain

No, that's simply not how SMTP routing works.

> No, that's simply not how SMTP routing works.

So you want some third party provider to be delivering mail on behalf of a domain for which they don't have even have the basics like DKIM and SPF set up, and hope for better deliverability than you could easily obtain with your own server?

Umm, no. The SPF and DKIM is something you set up in your DNS records, not they.

Your SPF record, created by you, indicates that the certain forwarding servers you have chosen are authorized to deliver mail for your domain.

When you change SMTP providers, you update that.

E.g. a year ago I switched from Shaw to Novus (two Canadian service providers). I edited my server's SMTP credentials to the new Novus server and user ID, password and changed the SPF record to bless Novus servers as being my delivery agents. That's it; mail was flowing through thew new configuration.

The ISP doesn't know anything about my domain or any of its DNS records.

Yes, they have better deliverability than I could obtain with my own server directly, because my server is on a dynamic subscriber IP which makes it a pariah in the world of mail delivery. Sending from it directly to mail exchangers world over is a nonstarter.

I could pay for some server in a cloud data center somewhere. What for? I have no issues with mail delivery.

Ok, SPF could work if your provider has a fixed and published ip range for its outgoing mail servers. (Is that common?)

DKIM though.. ?

But I get your point: it might beat a home server on a dynamic IP on deliverability. Both options seem troublesome.

SPF records have flexibility. It's possible to specify a domain name. Then any host which has an A record under that domain will pass.

In my SPF record I have novus.ca.

So I don't care what IP addresses Novus's mail servers use, as long as they identify as <host>.novus.ca.

DKIM-signed messages can pass through SMTP hops. I'm not briefed up on the details of DKIM, but to my best current understanding, the originating domain signs the body and certain headers (not all of them) with its private key. When the message passes through multiple SMTP hops, some headers get added, like "Received: ...". I believe, these headers do not invalidate the DKIM signature. The relays just cannot be messing with the body of the e-mail, Subject:, From:, Date: and such. SMTP relay is not like a mailing list repost.

I'm now looking at some raw e-mails with DKIM signatures. It looks as if the signatures plainly specify the names of headers that are included in the signature, via a field that starts with h=, listing colon-separated header names.

I setup mox a year ago with a new domain on a new server and delivered straight to Gmail within half an hour.

"The deliver-to-gmail test case passed; ship it!"

Pretty much, yes. Other providers are small enough (except for maybe Microsoft for business) that it's generally their problem if they accept less than Gmail.

If any of them hold the mailbox of someone that you or one of your users needs to reach, it quickly becomes your problem.

Now suppose you contact that server and complain about being rejected.

Wouldn't it be ironic if they respond like this: "We receive e-mails from gmail just fine; fix the problem yourself, or use gmail".

This is how self-hosted e-mail people throw each other under a bus and let gmail win, while pretending to hold self-hosting as a cherished value.

(They would most likely be right about having to fix the problem yourself, unless they imposed some locally authored and highly unreasonable/dichkeadish filtering rule. The superfluous rhetoric about gmail would be almost as obnoxious as their rule, though.)

hetzner allows outbound smtp by request. the process is relatively painless and quick.

Yes, but the process of getting Gmail, Outlook etc to receive your emails and put them in recipients' inboxes is far from painless or quick. An IP address with a clean history and SPF/DKIM/DMARC are table stakes, but then you get to play the "my emails are randomly dropped today while everything looked fine yesterday" game.

Anecdotally, we have hosted email servers for old games on Hetzner without issue, as the IP pool is generally not as popular with spammers given the time cost bringing up the server OS images. It is far from perfect, but generally performs well as reporting asshats on your local network block is easy.

Almost all cloud providers with dynamic-load ephemeral IPs will show up on ban lists eventually due to vulnerability scanners, bad spiders, and spam/voip drops. However, it is far more common for Spamhaus free tiers to quietly go sideways when no one is looking.

Gmail/Outlook have their own peer policies that serve their own business posture. Google does require administrators register in their clown system as a user to exchange email, but it is effective policy that adds nuisance cost to people spinning up 30 servers a day to spam people.

Firewall Rate-limits are effective on small single-domain servers. A modern email server in Go that is isolated from each user space greatly simplifies the possible setups. =3

At 15+ years of hosting my own email through multiple IP changes this has not been my experience at all. Hosting your own legit email works fine.

OK, well it hasn't been MY experience at all, hosting your own legit email with a 100% score on mail-tester, SPF, DKIM and DMARC does NOT work fine because Microsoft still ends up marking all your emails as spam, so maybe you could consider your experience is not universal and just because it happens to work with your IP addresses doesn't mean that's the case for everyone else? Jeez...

No one is “right”.

My experience is that Gmail accepted my emails fine... until one day it didn't. Then some time later it worked again.

I registered for their Postmaster Tools, which says

    No data to display at this time. Please come back later.
    Postmaster Tools requires that your domain satisfies certain conditions before
    data is visible for this chart. 
    Refer to the help page for more details.

The help page has no useful information. I suspect that I sent too little mail for it to register in their systems at all.

Outlook was even worse, and I just told my Outlook users to change providers.

Eventually I capitulated and got Google Workspace, and now everything gets delivered perfectly.

15+ years hosting email on the same ip space with strict security process. Numerous numerous numerous blocks, black holes, and spam routing. This was personal.

Worked for a company self hosting famous brand emails. They would get blocked too. Imagine telling the band manager of a famous classic rock band that their email to their label was being rejected due to being black listed for spam.. (cc’ing the managers team)

Stop fooling yourself, it does not work fine. If it did you would not rely on that google outlook or yahoo account

Perhaps you're replying to another comment.

That's commercial email. Of course you get flagged for spam. Use a service for that kind of thing.

Personal/private/family email can be easily self-hosted. You just need to know a few things to get it set up properly.

> At 15+ years of hosting my own email through multiple IP changes this has not been my experience at all.

At 25+ years of hosting email through multiple hosting providers, this has been my experience multiple times. To be fair, happening less often with DKIM et al, but those are relatively new inventions.

EDIT time is over. I don't want to be misunderstood. I am not claiming to send MASS emails and having them delivered without issues or anything. If we have to do mass emails, they are done with services that provide the GUIs for them etc. There's no way you won't end up in spam lists even if you sign up each invidiual email address in person yourself.

That's true sending email from my MS Outlook box to my own gmail. At some point, it comes down to just doing the best you can and not stressing too hard.

Getting a dedicated server with an ISP that does a decent job at keeping their IP blocks clean for email is about the best you can expect. Setup the appropriate SPF/DKIM/DMARC and get along. There's really not too much more to be done these days. Even the big guys don't always get along.

Follow the mox quickstart instructions and you might be surprised how successful and maintenance free it is.

I am sending and receiving emails on a small rack server in a datacenter for 40+ domains, and have had no real issues with deliverability. YMMV but I believe the reputation problem is heavily skewed against cloud providers such as VPS hosts more than anything.

I’m curious to know how you could know if any emails you send are getting silently dropped. Do you check with the recipient again and/or through other modes of communication?

I, too, run 4 email servers serving 12 domains overall, for about 11 years. I don't remember any email-related issues in the last 5 years.

One of the server sends and receives emails for the forum, sometimes up to 1000 messages a day. It was set up 5 years ago.

Maybe this is a serious issue when you use popular VPS providers/IP ranges, but I use smaller providers, and just don't remember any email-related issues everybody are talking about.

For me, email self-hosting as easy as installing mail-in-a-box (for sending+receiving) or just plain exim/postfix (for sending only), with proper configuration.

Almost all of the emails being sent from these services are transactional, so we would see noticeable abandoned user activities (i.e. confirmations, 2FA) or complaints from active users about not receiving emails.

We also have receipt tracking, which isn't perfect, but shows a >93% open rate.

We did have an issue delivering to a specific provider, but that was resolved by updating our DKIM with a more robust key length.

What you have is really great. Hoeever, if I had a small rack server in a data center, I wouldn't be able to call it self-hosted with a straight face, unless I had an uncle who owns a 60% share of the data center or something.

Been running my own mail server since 1999 or so. No issues.

You're grandfathered in. Fresh starts face an uphill battle with taint and reputation.

Been running a mailserver running Mox with a brand new domain on a brand new OVH IP for a year, no issues delivering to anyone.

> Been running my own mail server since 1999 or so.

Same.

> No issues.

Many issues.

> It's practically impossible to > send SMTP from your own IP address.

I haven't had any problem in that regard in over 20 years of running a mail server on an old PC, on residential ISP connections. SPF, DKIM and rDNS config seemed to keep all the big players happy.

Which just made me realise I don't even have valid rDNS anymore, but it still works.

https://www.mailreach.co/

> You can't run your own e-mail, or not entirely. It's practically impossible to send SMTP from your own IP address.

I beg to disagree, as I've been running my own E-mail and sending from my own IP address for [checks notes] the last 25 years or so.

Re mail deliverability. My experience so different than what you are saying that I take comments like this as regurgitating FUD at this point. Please do not do it. Even google is mostly OK with just spf or dkim. It really isn't that hard to host your own email.

I’m on an open source email list, where a lot of users self host their email. They have all the correct things done by the book. But gmail sends them all to my spam box, despite my continuing to mark them as not spam. Some even don’t appear in the spam box, despite other users on the list receiving the emails just fine.

Was your intention to prove the point about why it is important that more people host their own mail servers, or did you not realize you were?

I wasn’t stating anything moral, just disagreeing with the claim that you can do “just spf or dkim” and not have any issues.

If this email list does not rewrite "from" header, and modifies the email contents, that's the issue. Unfortunately, many still do. Such setup just won't work in the modern email world anymore.

It’s mostly Microsoft that is a problem. I’ve heard of a couple of cases in the past years where recipients used Microsoft’s services and never received emails from small self-hosted servers (where SPF, DKIM, etc were all properly set up).

If your client uses MS for email and doesn’t receive your invoices, it becomes a big deal.

I've had mild but inconsistent success sending to gmail with a perfect setup with 100% compliant dkim and spf, but Microsoft servers might be flat-out unreachable with no way to appeal:

https://news.ycombinator.com/item?id=35691618

In the end I set up a gmail account just to route all my outgoing mail through, with a whitelist of specific servers I know won't reject me for no reason (i.e. a few very small email services or friends who also self-host). Defeats half of the purpose but what can you do? There's nothing else I can possibly do to make my emails reach hotmail inboxes - I've exhausted all of their phony support channels and advice articles and clearly they just want me to go away and stop self-hosting.

They also want a PTR record on your IP to match your SMTP banner matching you hostname. Having an mx record for you sender domain also helps. Just sending form an IP address usually is tagged spam in my experience. Its weird their FAQ doesn't mention reverse DNS at all, its a very important step in having a good sender reputation.

The PTR match is a signal, but it's not required.

I send mail from several domains out of my mail server. The PTR record for that host actually doesn't match any of the forward hostnames.

I didn't say anything about it being hard; just that you may have to use some proxy for sending mail rather than doing it directly. This is not hard.

It does mean that you are slightly less than perfectly self-hosted, in some sense.

If your mail server is in a position that it can send mail directly to any mail exchanger in the world, rather than going through a forwarding host, there is the advantage in that it can use end-to-end TLS.

Reply on news.ycombinator.com