If you put some decent effort into making sure that you don't send spam, try to monitor if anyone thinks you send spam and react when someone complains that you send spam (and stop it), it works.
In my experience people telling these stories often do send spam, but they don't believe they do. ("It's not spam, it's a Newsletter. No, it has no unsubscribe link. These are people that agreed to be put on the newsletter by clicking on some ToS they never read, and they can unsubscribe by some arcane mechanism that we will make as complicated as we can. But we're definitely not spammers.")
But what is a problem is providing a good enough web interface, search, and so on.
However, one of my servers IPs is on a Microsoft blacklist since many years now. It sends <10 messages / day. I've tried every unlist form I could find, even called MS but it does not get taken of that list and they "won't disclose why". I'm routing SMTP to MS via another relay now :)
How do you do this? Could you share details on the setup?
transport.db:
hotmail.com relay:[relay.server.tld]:587 # and other domains
main.cf:
transport_maps = hash:/etc/postfix/transport smtp_sasl_password_maps = hash:/etc/postfix/relay_passwd
relay_passwd.db: # if necessary / not authenticated by IP relay.server.tld user:pass
The relay can/should rewrite the Return-Path to pass SPF. It's no problem for DMARC as the DKIM signature added by the initial server still authenticates it.
It requires manually adding domains of custom 365 installations to the list - at this size I do this manually, but should probably be automated "on bounce" or maybe even by a smart rule based on the MX record.
In Exim4 it's also possible to conditionally rewrite based on for example the recipient domain.
Well, I put in more than some decent effort, and I didn't get it "to work". I detailed my efforts here:
https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...
Please stop spreading falsehoods. If you were able to somehow get your own email server to deliver email to Gmail and Outlook, great, good for you - but stop pretending that anybody can do it.
Yes, that's probably true. I've been running my own server for 20 years now, and I guess that in itself helps with getting my mail delivered (apart from t-online, but who cares about them). At some time I also hosted some mailing lists, but I quickly abandoned that because that's a surefire way to get your IP blacklisted sooner or later. If you set up a completely new mail server, there probably is a lot of luck involved, and I wouldn't recommend it to anyone, at least not for your critical business mails. I pretty much keep doing it only out of nostalgia, it doesn't really make any sense otherwise.
A few years ago we had problems, but then I realized some of the emails sent from our servers had non-ascii characters in headers (subject, from, to) which caused email-providers to distrust our server. Using encoded-words syntax ("=?UTF-8?B?" + BASE64(text) + "?=") fixed that problem:
Most of the time, delivery problems were of my own creation. Like running out of disk space or accidentally disabling TLS.
Once in a while, Microsoft would start swallowing emails or Google would push everyone to use DMARC.
But overall, the experience has been very pleasant. I host my mails, I own my data. I am not shy of using Google, but my work is not defined by their whims. When Google tells me I ran out of space in my account I just delete stuff because I have copies of everything outside of Google infrastructure.
It requires some effort to maintain and understand, and I’ve had a few deliverability issues over the years but they are generally with niche providers. I’ve never had trouble sending mail to the big providers.
Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
>Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
I feel the same. I've had one or two hiccups but smooth sailing for the most part. I'm also happy to provide receipts that show how the recipient's mail server is responding when I send the emails. It's a powerful tool to say, "your mail provider is misbehaving, look!" They will wonder how many people tried to send them email that didn't get to them.
What ISP should someone choose?
I'm doubtful a default block would work, as that would even penalise the 'big boys' of email when they make basic network changes and piss off existing customers of both sender and receiver... Its easier and logical to conclude something without reputation yet is therefore sending too few mails to be useful to a spammer.
I've had good experiences with smaller ISPs (currently Mythic Beasts). In contrast, OVH was a poor experience.
I find that reputation (beyond the known "block-lists") appears more likely being tracked for the whole AS number, therefore a lot more to do with your "neighbours" than anything else.
What matters most is if the IP address they issue you has been blacklisted for spamming. DigitalOcean is fine but you need to check the IP address before you do the work of building a mail server. Some of their IPs are on a lot of blacklists.
If it's only on a very few you need to look into who's blacklisting it. There are some that offer a way to get delisted and make it easy, there are others that block pretty much every IP address DigitalOcean has (or large ranges of them) and they won't de-list anything within them. Many of those blacklists are managed overseas and not used much in the U.S.
No matter the ISP you should check the IP address they issue for a VPS before you build the email server.
My personal gmail account is full of spam and emails I do want from email lists end up the spam folder randomly.
That being said, now I monitor and auto-ban failed authentication attempts to smtp/imap (among others) and running the service is fairly low maintenance.
But the morale of the story is that you are only one weak password from one of your users away from your mail server getting blacklisted as a spam server. So while I think it is fairly easy to run a personal server, running one for a small organisation is another matter.
That's about as easy as it gets but it still requires some work and you need to check the IP address DigitalOcean issues to see if it's blacklisted before you set it up.
Google makes it easy to get whitelisted. Microsoft email services (Hotmail/Outlook) are a pain though. I tried to get through their process but got nowhere. Other services I had to submit a request to get de-listed. So it does take awhile to go through all that.
Still, I prefer that to hitching that wagon to a 3rd party provider like Google, or any other.
Before I set mine up the 1st time I'd been screwed a few times by 3rd party providers. The last one, I can't recall which, but it was either "MailChimp" or whomever bought them, that I'd configured an app to use and almost as soon as I'd released it they announce they'd been acquired and I would have to use the new services APIs, and of course they cost more, and their services were geared towards mass mailing, and that's not what my apps do, and their API sucked for my needs.
It was about 12 years of dealing with 3rd party bullshit that motivated me to set up my own email server.
If you just want to fiddle around with one to get a feel for it Mail-in-a-Box is a good place to get started: https://mailinabox.email/
I've been running my own mail server since mid '00s. Initially hosted with one of West coast Canadian colos and subsequently moved to an EU colo. Had some deliverability issues with Outlook and Yahoo, but these were episodic and rare even though I set up DKIM only last year and have been running with just SPF and DNS/PTR before that.
I know at least a dozen of others with similar setups and timelines. But we all use dedicated colo'ed boxes on IPs from clean netblocks that weren't previously used for shared hosting. I strongly suspect that attempting to run a mailserver on Digital Ocean, OVH, 1and1 and similar mass-hosting providers will not go well. Just like it will be an uphill battle to run it on a residental IP.
What did I generalize, exactly? Parent poster was claiming that anybody can set up a mail server with good deliverability - that's a generalization. I said good for them (acknowledging they managed to make it work) and said that I also tried and couldn't make it work - therefore, clearly not everybody can make it work. Did I not argue against generalization there?
I run my mail server on Linode, no issues at all.
I do it as well and apparently so do many others.
Not sure why it seems problematic for some, but it hasn't been an issue at all for me.
I've run a hobby website for about 15 years that does not even have a newsletter of any kind, and includes "stop sending me emails" in each transactional email (all users are double opt-in verified), and password resets are still not delivered half the time to gmail addresses.
Frankly, I'm shocked you've never been arbitrarily blocked and I find your insinuations offensive.
The last time I was getting blocked it was the solarwinds fiasco where their internal mail tunneling/forwarding and filtering setup broke all DKIM and suddenly solarwinds users like NOAA.gov were rejecting me and adding me to naughty lists. There was no fallout for the megacorps and their broken setups. There was only damage to independent mailserver operators doing the right thing.
I've run my own email for decades and I've designed and run some pretty big commercial installations.
As a small provider, you run the risk of existing in a netblock used by other people sending spam. A small co-op I ran encountered this problem once. They were operating on the cheap and while they weren't sending spam their neighbors had been.
Even as a large provider at a billion dollar company, figuring out delivery issues is a huge pain and generally not worth it. There are unofficial professional postmaster meetups around the bay and these can be helpful in getting escalation contacts to fix issues, but even with entire teams of people dedicated it's a lot to handle and is usually worthwhile to outsource the work to other companies who already have these types of relationships established.
Unlike the person to you're replying to we had no issues with Google or Microsoft (once we did the requisite things) - it was Yahoo (and the people they provide email for) and then multiple mid-size organisations who used IP based block lists. At one point our mails were being rejected by our local NHS trust, the London Fire Brigade and a mental health agency we make referrals to. None of this was complicated to resolve but it was energy that could have been better used elsewhere.
I'm not usually part of the "let's go cloud without doing any cost-benefit analysis" movement but with email delivery I was happy when we could go back to routing via Office365 again. If a recipient decides to ban Microsoft's IPs that's usually going to be a bigger problem for them than me.
If you can have a high degree of confidence that no outgoing smtp spam traffic has ever been emitted from any of the other IPs adjacent to where you're hosted, the opaque blacklists of the big mail receiving providers (gmail, etc) are much less likely to consider your legit traffic as spam.
I've been running mailservers using free software for 20 years. I've run two for personal use, and several for groups like companies. In the old days, you could indeed throw up a server, and provided you don't spam, and you're not in a bad neighbourhood, outgoing mail would be accepted.
In more recent years, my experience has been that it takes time for a new mail sender to be acccepted; could be a year or two to build reputation. That's assuming you do everything right.
My personal mail, by the way, has been on the same domain since about 2001. I've quit running a mailserver now. My small ISP runs a setup that's basically what I would have built, so I use that; the support is excellent. But it's still on the same domain.
Last company I was at ran their mail on their ISPs mailserver. The ISP got taken over; service deteriorated, to the point it became unacceptable. So I built $EMPLOYER a mailserver; it took me longer than I predicted, because the bosses had all kinds of finicky requirements (don't they always) that I had to figure out how to provide after the fact. But that "artisanal" server beat the bejabers out of the ISP system; it was fast, reliable, and when anything went wrong I could fix it - which that ISP couldn't.
Also running my own servers for personal and business, and working well.
But when we tried to use one of the large VM providers the experience was much less reliable. Despite ensuring the IP was not on the various block lists etc. mails would be accepted and silently discarded by recipients ISPs, perhaps due to the level of abuse of these IP ranges.
Yes. I do get that impression from most complainers.
I send from my own domains, and if I sent it, I wrote and addressed it personally.
If I send an email to a corporation’s customer support, or to a distant relation, or to an open source mailing list, and I don’t get a reply, it could be a delivery problem - or it could just be that they didn’t decide to reply.
At work I ran email servers professionally and with good deliverability for years. My own email server was arguably longer lived than those at work, just much lower volumes. IP block was clean, DKIM, SPF, rDNS, etc. all setup correctly.
I thought I had no deliverability issues. I interacted with mailing lists regularly, the odd email to friends and family and I was firmly in your camp until I had to deal with a death in the family.
I think this was shortly after Microsoft BPOS became Office365. It became very very clear very very rapidly that to certain orgs I just wasn't hitting the inbox. And there was jack shit I could do about it. That was the end of my mail server, and it's certainly got worse over time.
This is a popular opinion on HN but it doesn't seem at all inline with reality. Email isn't exactly a real money maker for anyone. And the amount of email spam and abuse is immense. Filtering out most unknown providers is unfortunately extremely effective. Almost all spam wiped out with a simple check.
Maybe the ideal solution would be to let you link your custom email domain with a google account so you can have your google account vouch for the legitimacy of your custom domain. But even then, some of the time your email server actually is just blasting out spam without you knowing it.
There's a tendency to perma-reject e-mail coming from "not seen before" domains despite the e-mail passing FCrDNS + SPF + DKIM + DMARC validation, which makes it difficult for private e-mail server users to get through to people.
The unspoken "you silly prick" gets louder every time this happens.
The big players all have a process and followup within days.
I’d argue that the vast bulk of email is sent from dedicated providers like Sendgrid which are built on the same tech that might be found in any given ‘artisanal’ on-prem service.