The big players (Google, in particular...in case it's not clear, the bulk of this rant is leveled at Google; Microsoft is an offender on perhaps 25% of the level, Yahoo just doesn't seem to care) have ZERO interest in "playing nice" with small e-mail servers. My experience running my own e-mail for almost twenty years then finally throwing in the towel and migrating to Fastmail not only proves this for me, but is, according to other sysadmins I've talked to, quite common.
I did everything right when hosting my e-mail. I did it for myself and maybe four others. The volume of mail going out of my server was so low, I could read the outbound mail.log for myself, and routinely did. Every domain had SPF, DKIM, and DMARC records. The outgoing single IPv4 and IPv6 addresses I used had perfectly-matching forward and reverse DNS lookups. I had a script that checked every blacklist I could find every 24 hours. My IP addresses were allocated from my (professional) colocation provider and SWIP'd to me with valid contact information. My IP addresses had been unchanged for seven years and had been stable for ten years prior to that change. My colo provider had a draconian non-spam policy. Nobody ran an e-mail list or anything. I've used the exact same domain for e-mail for over twenty years.
Still, every couple of months, Google (sometimes Microsoft, but far and away most likely Google) would decide to simply stop delivering messages from my five users--remember, including me--to Google's users. The messages would just...disappear. Sometimes, Google would stop delivering the messages mid-conversation. Replies would be going back and forth but then my reply to a Gmail user would go to /dev/null while their messages to me would still arrive. Most of the time, the mail would go to spam (even with that little yellow tag meaning "verified sender" or some such) but 20% of the time the mails would just disappear.
Worse, the destination would say "250 OK" to every e-mail I sent them so I had no way of knowing that the messages had been sent to the big bitbucket in the sky. I got annoyed.
I signed up for Google's Postmaster Tools and Microsoft's similar offering. Microsoft would at least show me telemetry but it wouldn't do any good. Google? Google would proclaim that I wasn't sending them enough e-mail so I couldn't show up on the dashboard.
Finally, after three years of fighting this, it came to a head when I was, at all places, at a boardgame meetup during one of these "blackout periods" and got introduced to a Google SRE who worked "adjacent" to Gmail. "Oh, hey, you should talk to techsupporter, he does e-mail too!" was the opening line. We talked, I ranted, and he actually said he'd try to help. He escalated internally and the word came down from on high on their SMTP team or some such: "domain [mydomain] has insufficient reputation."
What did that mean? Apparently, their system--at least at that moment--thought my domain was "too new" and they tilted towards just binning the e-mail. I got red, saw stars, then gave Fastmail $390 for three years of service on three e-mail accounts and moved everything to them within five hours. That was the last straw, the final injustice. For me, the "decentralized Internet" died on that day.
Why was I so pissed? Observe these dates from WHOIS:
Domain Name: MYDOMAIN.ORG
[...]
Creation Date: 1997-03-10T05:00:00Z
Domain Name: GOOGLE.COM
[....]
Creation Date: 1997-09-15T04:00:00ZThere is a specific reason I recommend doing so: the reputation of the sending IP address almost always overrides the reputation of the sending domain.
A smart host operated by a third party, such as Google: https://support.google.com/a/answer/2956491?hl=en , would have solved your blackholing issues because it's the third party's job to maintain extremely good reputation of their IPs sending mail. Most importantly, they have high volume of legit emails, and volume is key to maintain good reputation. Finally, even if you have a low-volume domain, your emails will benefit of the good rep of the smart host, and thus would not be blackholed.
I will repeat it again: if you host your mail server, you definitely should also use a smart host, and all the spam/blackholing issues disappear.
Google also rewrites the "From:" address to your Google account's address, unless you jump through some hoops to add it to your account. This has become more difficult recently, and needs to be done for every address (which is an issue if you use single-purpose email addresses).
Many things wrong with this sentence:
• Don't like Google? Then use any of the hundreds of other smart host relays: Sendgrid, OutboundSMTP, etc
• The smart host can't read your incoming mail (typically more sensitive than outgoing.)
• Google, being a mail provider, can usually read your email anyway, because all your conversations with Gmail users end up in Google's hands.
«Google also rewrites the "From:"»
Google has multiple smart hosts offerings, but the one I linked to does NOT rewrite the From:.
This. I go to a lot of trouble to follow all relevant RFCs, SPF, DKIM, DMARC, and TLS. My intent is to help other servers know I'm transmitting in good faith, I'm hopes I'll get the same fair shake.
If it's still not good enough for you, please at least let me know that fact, with a message I can investigate. At the very least, don't outright lie, which is what I take a bogus 250 to be doing.
This was your actual problem. The large providers (gmail, outlook, yahoo) and appliance vendors (ironport, etc.) are all now treating "unknown" senders as spammers. If you don't send a large enough volume of emails, you will never make it into their reputation tracking lists because "it's not worth it". End result, you are considered unknown and will be treated as a spammer forever.
It is a huge organically grown cartel where the protocols work but you cannot enter a small player, no matter how well you behave. It is a shame, but there's nothing you can do about it :(