Microsoft Silently Dropping Emails – a Sad but True Story (2020) | Hacker News Reader

Microsoft Silently Dropping Emails – a Sad but True Story (2020)

nerd-quickies.net

221 pointscatherd108 comments

I had this problem repeatedly with both Google and Microsoft, though never at the same time and never, so far as I could tell, for the same reason.

Everything was set up properly. The domains I hosted had IPv4 and IPv6 addresses with forward and reverse DNS matching exactly. SPF was set perfectly, according to multiple test sites. DKIM was always used and always validated, again according to multiple test sites. I even signed up for a DMARC endpoint account at report-uri.com.

Didn't matter. Messages would be spontaneously rejected, sometimes in mid-conversation (this happened to my wife quite a lot). Other times, the remote end would claim to accept the message and then it would disappear into the great bit bucket in the sky. It would be like this for a day or two then spontaneously go back to working.

I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).

> I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).

Can anyone speak to what a provider like Fastmail does behind the scenes that guarantees their delivery? Is it the threat to sue/kick up dirt/raise issues with the appropriate regulatory body if perfectly good emails don't go through? Merely having a seat at some sort of regulatory body or committee?

From what I can tell, there are two main reasons companies like Fastmail don't have these problems. The first is that good email providers spend a lot of time maintaining the reputation of the IP addresses they use and shut down bad actors as quickly as possible. The second is that the more people that use the service, the higher percentage of good quality emails being sent, reducing the effect of the occasional bad actor.

I'm self-hosting my mails. Microsoft expects you to send enough mails to consider your mails not as spam. One thing Fastmail does that people who self-host can't is: sending enough mails.

I don't know what other measure they may take but this alone probably goes a long way.

And yes, this is fucking frustrating. I'm not going to send thousands of mails per months just for the sake of it and don't know how I should do that without spamming. This is so fucked up.

This is definitely something I've never heard of, thanks for mentioning it -- I had no idea there was such a consideration (or even that it would be a good way to filter out scammers)

I think it's likely that they directly share an allow list of each other's server to form some sort of network of email providers.

Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms? Sure, you can put up the server, but most people are on one of the big providers, and chances are they've got your IP range blocked, because nearly every IP range you might be using has probably been owned by spammers at some point.

Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general. In theory, it's an open ecosystem anyone can jump into. In reality, unless you have a lot of energy/time/money to get around the big providers banning you for "spam," it's not going to happen.

To be fair, I was also surprised to read that anyone was still using Hotmail, so maybe I just don't have my finger on the pulse of the hip new generation.

I spotted someone in the wild using a netzero email address this month as their day-to-day.

"Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?"

Not for me and loads of my tiny and not so tiny customers. Just keep your nose clean and follow the rules. This is in the UK, elsewhere ymmv.

FWIW, not for me either. I run mail servers for a small(ish) organisation.

In fact, I'd say it's getting easier, not harder. There was a time when fairly arbitrary rules were applied (like "thou shall not send email from a dialup IP Address). They seem to have mostly gone now, and when they are still in place you can just self report via a web form you are not being naughty and it's all good.

But you do have to keep your nose clean. Do not send SPAM, under any circumstances. Sounds easy, and indeed it is easy if you are just running a mail server for an organisation where all mail you are sending is generated by people within that organisation. But if you are doing something that involves relaying emails, eg something like running a mailing list, it becomes harder.

To give an example - I run a mail server for a club of maybe 20 people. All email is strictly policed, eg you have to sign up, all sign ups are reviewed by a human, including any email asking to sign up. All good right? Nope, because the club has to have at least one email address (say for the secretary) that does not involve signing up. Those emails are all moderated of course - but you are notified of emails waiting for moderation via email. Those notifications contain snippets of the original. If one of the moderators has a GMail account, those snippets triggering SPAM filters can get your server banned by google. The solution is to run a imap server for secretary@ and get them to fetch it manually. Bypass Google for all relayed email in other words.

So you have to follow the rules and it's not always obvious what you have to do to follow the rules, but it ain't impossible, and in fact once you know what the rules are it isn't even hard. Any capable sysadmin could do it.

It's still doable. I self host and have managed to switch IP's providers and so forth without getting into trouble. It is, however, critical that you make sure you don't get an IP that was recently used by a spammer, and it's a lot of work to get everything up and running. Warming up an IP is difficult when talking volumes that small.

Do you have any tips on warming up your IP? I’ve been going down the self-hosting rabbithole again after years and just set up a server. I’m about to set up a mail server in the coming weeks and checked that my IP is not on any blocklists, but my normal email traffic is very low. Should I automate sending an email to my accounts on major platforms every other day or something along those lines?

Send authenticated mails. Send an email or two to your own Hotmail/Gmail address each day (or at least regularly), and make sure to interact with the email (read/reply/forward/put in folder/etc.). Personally, I use Hotmail for reading my shopping lists.

If you end up in your own spam folder, "Not junk" is your friend. If you don't get through at all, do the escalation dance with Microsoft until the situation improves.

Once you reliably hit your own Hotmail inbox, start sending mails to A FEW others as well. Verify that they receive the mail, and ask them to add you to their trusted sender lists, press not junk, etc, and keep bugging Microsoft if they just discard your emails silently. Once you have a few people doing that, reputation should improve quickly.

Your are still allowed. I self-host personal mail in a VM and still can send mail to gmail.

The problem is that big players (Gmail/MS) care about an anti-spam false positives only when they affects a relatively large fraction of all messages. Small mail servers account for a small fraction of all messages, so big players don't care about them at all.

I will not be surprised if they have SLI/KPI which allows them reject by a mistake millions of non-spam messages daily because it is a small percent of all messages they process.

> Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?

Well I've been self hosting my emails for 10 years now, and my emails still go through. I'm sure there must be cases where you face issues, but not my in my personal experience.

Only thing I've seen once or twice are websites with "validation" rules like /.@(gmail|hotmail|yahoo)./

I have also been self-hosting my email for almost 20 years.

Until a few years ago I never had any problems, but during the last 3-4 years I had sporadic problems with messages that either were not delivered or they were delivered with a very large delay, e.g. a day, or they were delivered only if I resent them 3 or 4 times, at some of the large providers, like Gmail.

It still works fine most of the time, but when problems happen they are annoying because there is no way to know that something happened, unless I have an alternative way to ask the recipient, e.g. by phone.

In any case, now I always check that at least the final destination e-mail server has accepted the message, which should have been enough to guarantee the delivery, according to standards, but nonetheless this is not true any more.

Yep. For important emails, I tail the exim mainlog. But in six sigma cases, it just works or I get to know that my mails were not delivered days later.

Is big evil tech at fault though? I'm sure the vast majority of users asked Microsoft and Google to do something about spam.

Iirc at its height 90% of all email was spam.

I'm pretty sure 0% of MS's customers asked them to silently drop emails without either recteting the mail at sumission time or dropping it in the spam foler.

I well remember the times when I set up sendmail and saw emails immediately sent in Netscape Mail instead waiting for minutes. This was using a 32kB dialup connection.

Was like 1997. Needless to say it was then sendmail struggling to get the mails over dialup. At least my single-threaded Netscape was responsive.

Worked until 1999 or so. Then spam became a thing and more and more mails were rejected and I folded my setup.

>> Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?

For some time seemingly legit mail from Azure Cloud Services also got flagged by Gmail. If that was the case -- I thought -- what changes do I have for my own hosted or non-major email provider? That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.

> That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.

Do you realize people like you giving up is precisely why these evil corporations can get away with what they do? Together, we small providers are strong. But if some resistance fighters drop the struggle and join Dark Vader 's side, we are bound to loose.

Maybe we should start a campaign. Give a one-year notice to those big email provider, in partnership with smaller email providers, saying they have one year to comply with email standards or we'll start boycotting and promoting alternatives as a federation of free/open email providers?

We could start doing like the comment on OP does. Send a rejection notice with a link to some list of all the various blogs documenting how big providers refuse to follow the standards.

> My other thought was to have my Postfix server reject all mail from Microsoft servers, with a message requesting the sender to use a different mail provider and pointing to this article.

I've often been tempted to do that. However, we would actively be hurting our own users who would be incapable to receive mails from such servers. We'd in fact be hurting the very principles of interoperability we're defending in the first place.

Adding an auto-reply or an email footer mentioning that their providers actively hurts the email ecosystem and linking to alternatives would be a good compromise in my view.

Along with a libre email provider federation, that could go a long way...

Some companies reject to communicate to you straight away if you use GMail and don't run your own company EMail. On the other hand some do the opposite and demand you use GMail.

After testing - looks like new sign-ups get a choice of @outlook.com or @hotmail.com (although users with previously available/legacy domains like @hotmail.co.uk retain them).

Apparently the service as a whole has been based on o365 since around 2015 according to wikipedia.

> Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general.

But isn't the federated nature of email the only thing which prevents the said monopolies from taking over email completely? If it can be proven that X giant is blocking mails from Y small email company then they're in lot of trouble at least in EU.

That's the reason chat and everything else is a lost cause; Can't send a message from Messenger to Telegram unless you use their bot API to build a specialized application[1] which is a downgrade from XMPP like interoperable protocol.

[1] A product from my previous startup which allowed communication between Messenger <-> Telegam <-> Viber <-> LINE for dating - https://hitstartup.com/about/#FindDate

Lots of people self-host their email, it works fine. The worst problem I've encountered is graylisting delays. Sometimes Gmail will throw a fit, but that's true whether you're originating from @icloud.com or from @yourtinydomain.com.

A couple years ago I did some superficial analysis on the domains of the registration emails our customers were giving us. I don't remember the details, but IIRC only about 80% of them were connected to one of the big providers (google, apple, qq, microsoft, yahoo(!), etc).

I knew about about but only found out about two years ago, and I'm not one who is bad with tech either. So how common of knowledge it is is debatable I would say.

I'm currently having the same experience.

Unlike virtually all other email providers, Microsoft does not cooperate if you follow standard email practices. You can follow all their instructions, sign up for their monitoring tools, submit issue reports, try to build "reputation" and if you are self-hosting they will still have a high likelihood of silently dropping your emails after their server tells you it got them, or putting them in the recipients spam even after being marked as an approved sender.

Unforgivable. Literally destroying the decentralized nature of one of the last few "decentralized" systems that exists. For shame

“Email is so unreliable. Have you tried Teams?”

Just ignore that it’s Microsoft breaking a legacy standard to advantage their other products that compete against email. I think they should start fining companies in stock. Dilute the shareholder value by 5% and see how quickly things change.

I like this, let's start a club

> Teams

Moving back to a full MS stack after 15ish years away from it is so disappointing. I’d assumed things had got better.

Their previously good products are now some sort of split personality monster that sort of work together but mostly just exist near each other and have a common logo. Teams is a mushy world between the distant parts.

Go on, try change a footer on a “Word” document or move an image. Try and copy a document and change its contents and re-upload. It might work, and it might work most the time. But when it goes wrong it’s devastating.

That's basically my experience administering Microsoft 365 at my last job.

It's almost easier centering divs in CSS. And that's saying a lot... Lol!

Yep. My Dad's company uses a small hosting provider so he cannot email me @live anymore.

Try adding him to your contact list.

Try to email your dad first. If you're the one initiate the conversation, then your dad will be able to reply to that same thread. After that, add him to contact list once he responded. The point is to make an attempt of your self initiate the connect to that server.

Doesn't always work. I've seen Outlook 365 or whatever it is mark a reply as spam. It's rare, but apparently not impossible.

Have you considered switching to a better email provider than @live then?

>Biggest problem is, that Microsoft always blacklists the whole server! If only one of my customers does something not in line with the secret rules of SmartScreen, emails of all other customers are blocked too. IMHO this is just unprofessional. Why not just block the domain under suspicion and send the adminstrator a message (maybe as part of the JMRP)?

The reason for blocking IP originates from a culture set before DKIM. Any IP could shoot out emails for any domain. You don't want to blacklist a domain for an email it did not legitimately author.

However, in a post DKIM world, where adoption is above 80% this could be a feasible solution at least for emails carrying a DKIM signature. The domain owner has provided proofs that the IP address is being used by them to send out emails. Flagging emails from a domain and IP could be blocked while allowing other domains from the same IP to operate.

This is anti-competitive behavior by definition. What a farce.

As someone who worked on reputation at Microsoft, it sounds like a bad case of the right hand not talking to the left. Outlook SmartScreen judgment should be available to tier 3 support along with the IP block list check as a primary investigation step. The author should not have needed to go through tier 3 on two tickets before escalating to someone who had visibility into SmartScreen judgments. Hopefully this blog gets some publicity and Microsoft support amends their investigation process, as I’m sure the author is not the only person running into this issue.

All opinions are my own and not that of Microsoft.

I've worked extensively with Microsoft support on deliverability issues, and I don't think it's a case of the right hand not talking to the left, it's a case of support wanting to close out issues as quickly as possible due to misaligned incentives. Microsoft has zero incentive to help small-time senders like the OP, since recipients are much more likely to blame the OP for delivery issues than Microsoft. Plus, Microsoft has no way of knowing for certain whether the OP is legitimate, so if they choose to help them, they might accidentally help a spammer. Therefore, Microsoft support is incentivized to simply ignore complaints like the OP or give them the run-around, which has been my experience.

The only part of the article that surprised me is the fact they actually got the issue escalated. I've never succeeded in doing that; talking to Microsoft support is like talking to a brick wall.

> The only part of the article that surprised me is the fact they actually got the issue escalated. I've never succeeded in doing that; talking to Microsoft support is like talking to a brick wall.

I’ve found that the best trick with Microsoft is: money talks. If you pay them for support suddenly you should have a much easier time getting things done.

Obviously paying MS to fix a problem they caused is abhorrent but sometimes you gotta hold your nose and do it.

> I’ve found that the best trick with Microsoft is: money talks.

It depends on the quantity. MS365 Business Basic/Premium customers are basically testers for the enterprise offerings. Look at release cadences for the different tiers and it becomes obvious.

> talking to Microsoft support is like talking to a brick wall

I once had DKIM keys get out of sync with DNS in Office 365. It’s supposed to be handled automatically and that should never happen. Support was as useful as a baby trying to do calculus. It took me 3 days to get the keys rotated and after I understood how it works I realized they should have been able to help me mitigate it right away (dkim keys can be rotated while disabled).

I had exactly same experience as author, however I didn't have enough perseverance to go that far. Changed IP and it worked.

Not sure how things are organised in microsoft, but it's one of worst support ever happened to me

Does Smart Screen filter email originating from within the organisation that sends it?

My organisation used to self host everything -- wonderful times; everything worked, but with a HTML 1.0 crappy web interface. About four years ago they shifted to Exchange (urgh -- suddenly IMAP errors and some of my message-read state information became lost. I also noted the massive decrease in my Mac's battery life as Mail seemed to have a lot more work to do -- but I digress) and about two years ago self-hosted exchange became O365.

They recently started pushing harder into moving everything into Microsoft's cloud, and my quality of life has massively decreased -- the first time I was driven insane by it was when my colleague's human-written emails to me ended up in my spam folder; and vice versa when I emailed six undergraduate students to set a date and time for a tutorial. I turned up, they didn't, as I'd been shadowbanned from sending a five-line message to recipients within the same organisation -- messaging them all together got it dropped, sending it individually without the CC was fine. I complained and got given the run-around for a few weeks with no answer. The response of my local "IT people" was "not our fault mate".

The other thing I particularly hate about Microsoft 365 is that all the user-facing error messages are "Sorry, something went wrong". I'm not an idiot, please don't treat me like one.

The problem, of course, is that Tier 1 support are outsourced and a huge part of their evaluations are how many tickets they close. Escalating hurts their numbers, so it's hard to get cases escalated, especially if you're not a paying customer.

> All opinions are my own and not that of Microsoft.

Having to write this is so ridiculous. Better just use a throwaway and be yourself. Or is this PR?

No, PR never admits to the management problems that are endemic to the company.

PR will mention a minor technical problem that has been solved, nothing to worry about.

> SmartScreen® Filter technology is always adapting and learning more about what is and isn’t unwanted mail, it is not possible for us to offer specific advice about improving your mail content.

Translation: "The filtering is based on machine learning. If there are false positives, too bad, even we don't know what's causing them. Try to look less like spam or something?"

They should be liable for their shitty ML algorithms. It’s extra detrimental to small businesses who get their email dropped for no good reason. Silently dropping email is malicious IMO.

ML is the software form of cheap plastic crap that quickly breaks.

Oh hey, Microsoft is breaking another open decentralized standard again as they are pushing their own proprietary alternative. But let's keep pretending they changed.

Speaking as someone who has been running MTAs for more than 15 years - Microsoft has been breaking e-mail for years. They are an order of magnitude more awful than anyone else, and no one should have their e-mail with them for any reason.

It all started with top-posting, and went downhill from there.

This is why IT should unionise, or form a guild.

Imagine if this was turned around, and simultaneously a huge chunk of the world instantly black-holed Office 365 outbound email.

Then in response to the roaring noise from Redmond, simply send back their canned response of "Not qualified for mitigation. Our investigation has determined that the above IP(s) do not qualify for mitigation."

Do that for 24 hours, then 48 hours, and so on until they smarten up.

I guarantee you they'll have their CEO breathing down the necks of every moron in the spam filter team until they fix it.

Until then? Nothing will happen. Nothing at all. We'll complain on YC News, we'll complain on /r/sysadmin, and a bunch of small businesses will go out of business. Microsoft 365 will continue to grow to consume the world. Email will be rebranded "Teams Mail", and as far as everyone is concerned that will make perfect sense.

Either we act, or we live with the consequences of inaction.

LOL.. watch what happens when you try and suddenly the government labels you as a domestic terrorist group because they are so heavily invested in Microsoft, and it actually helps the government to collect intelligence to only have a few approved conglomerate email providers.

"It's only terrorism if the little guy does it to the big fish. The other way around, we call that capitalism."

Collectivist action is great, when it's great. But with it comes all the usual ossification problems. (For a collective to have power others have to give it to them. Either the collective manages to secure a monopoly, like the various professional licensing/accreditation bodies, legal bar association, medical residency programs with fixed number of slots. Or stakeholders - companies, professionals, regulators - have to join the collective and give up at least some of their independence during the process.)

Yes, email is kind of important. But it's also pretty shoddy. It always has been. (Confidentiality, integrity, authentication, signed receipt of delivery. All are important for regular business correspondence and by default all were [and still are] hit and miss with email.)

This practice of MS of not accepting email might be abuse of market power. But they can easily say their clients signed up for Outlook to have exactly this protection. What would the courts think? Who knows.

Had the exact same experience when I worked at a non-profit: we had a self-hosted Postfix server, and our subscriber emails would get silently dropped. This only ever happened for Outlook.com users, and we had set up SPF, DKIM, and DMARC long ago. We got the same run-around with Microsoft support, and eventually we had to give up and move to using Google's SMTP Relay service.

It’s tacit collusion between the big providers to force all businesses onto their overpriced cloud services. One Office 365 user costs the same as 50 users at a shared host.

I run an email forwarding service https://hanami.run and I used to had this exact problem.

They had a dedicated page at https://sendersupport.olc.protection.outlook.com/snds/data.a... that supposed to be used for these delivery issue. I tried to submit multiple requests from there.

Even IPs that aren't sent any email in last 3 months.

Then one day the problem just disappears and I was able to send email just fine. Till this day, I still don't know what happen.

Another similar service is apple icloud, which I think even much more worse than Hotmail. At least, hotmail give you a portal that you can see some data.

With the announcement of apple icloud email service, I imagine people who are self-hosted email gonna have a lot issue with them.

> With the announcement of apple icloud email service, I imagine people who are self-hosted email gonna have a lot issue with them.

What announcement?

iCloud has had a E-Mail service for over 10 years now. iCloud was doing email back when it was called MobileMe.

They started to introduce support for custom domain. That means many will switch to them, for the convenience of the ecosystem.

So that means one more awkward email providers we have to deal with :)

Ah understood.

Apologies, I misunderstood what you were trying to say.

Thanks for clarifying :)

I started getting this on our transactional mail relay some years ago, hotmail responded with "Not qualified for mitigation", therefore instead of spending pointless effort dealing with these miserable jollems i used o365 to counter it (my organization was already using o365, which also never did this S3150 thing to me) by:

Adding a relay user to the o365 then configuring all the domains i needed to send from on the o365 tenant (without changing the MX records naturally). (Using a normal o365 user account to relay is possible for me as our typical customer base is enterprise and education, not random end users) Adding distribution lists as the from-addresses that i needed to send from and gave send-as rights to the relay user, compiled a list of hotmail/outlook domains (which i've had to add to over the years but it is relatively stable for me by now), then configured the outgoing relay server that does all the dkim-stamping to route all outgoing mail to outlook/hotmail domains through o365.

When i see a S3150 in the logs i just add another domain to the list, i should probably make this pre-emptive by looking at the target mx records before sending, as its not trivial to make exim resend something that ended in a 5xx.

All other mail providers have removed blocks by simply asking, not hotmail. Also all our newsletter-like mail goes through separate services, these are all transactional mails (meeting invites, user account creation, password resets and such).

I have had the exact same experience. Even put the senders on your whitelist and it makes no difference at all. It means that Office 365 cannot be trusted for handling mail. When it comes time to renew, after two years of putting up with this, I am moving off it.

Even the user side of Microsoft's email is confusing and prone to loosing emails. O365 defaults or at least the recommended settings will just quarantine emails without notifying anyone. I guess Microsoft doesn't trust users to not open emails in the spam folder?

I think I have it configured now to alert the recipient and me on quarantined emails but there are at least two relevant dashboard (Exchange and umm Security Center? who knows what the current names are today) that probably have like 35 submenus and 1000+ settings. Running my own email server would be easier.

Another thing, a few months ago I noticed a couple incoming emails were not quarantined/blocked but also not delivered. I don't think the issue has reoccurred since but it's just crazy that emails are just lost sometimes.

Meanwhile, I get so much spam on my hotmail account (from obvious spam addresses too) that I’ve had to switch to gmail. It seems both the false positive and false negative rate is sky-high.

Governments don't like self-hosting, because they need to ask for data every time they need it. For big hosting providers like Google, Microsoft, Fastmail and others they have mails already. So there are incentives to make it harder and harder to self-host...

Had the same problem as well when sending an email to a client of mine who used Hotmail. Had had successful communications with them in the past 1.5 years then suddenly, bam. Was put on the "S3140 block list". Had to use my personal Gmail to send emails to them. Never emailed another person who used Microsoft email services.

Was told my email server was sending spam by them (as people had reported emails from the IP as unwanted), so I did a full security check (using logs, trying to find holes, etc), and upgraded software. No problems that I could find.

After lots of back and forth, they said my IP was blocked as it was "suspicious" and they unblocked it (how kind of them). The process took a long time, which I could imagine could be devastating to a small business (luckily my client understood as I made sure to tell them how bad Microsoft is).

Unrelated but if the author is reading this please get rid of the sticky header on that site. It is beyond annoying.

Had this same issue many times sending via AWS (SES). Spent dozens of hours on it with no help from Microsoft, ultimately Amazon had to sort it out with them.

Silently dropping mail (not even to the spam folder) is a maddeningly cryptic action on MS's part, and has squandered massive amounts of developer and system administrator time.

If you're in a position to sway anybody from using Outlook.com, Hotmail.com, or Msn.com or Live.com I highly encourage it - MS has made a lot of lives difficult with their email policies, including NerdAdmin's here.

I had to stop using Office365 for business because it kept doing this same thing. I disabled every spam, and protection I could find, and enlisted the help of support. Some legitimate email would simply never arrive and they couldn’t explain why. It seems there are layers of protection, some with no logging (available to the tenant admin) to troubleshoot, it became a huge waste of time, negating the purpose of using a cloud service. In the end I moved to a different service provider and never had the problem again.

On the opposite side, I found Microsoft Azure to be one of the biggest sources of spam on the internet. I have blocked them completely on my private mail server.

This sounds similar to my experience with Microsoft based email (Outlook, Hotmail, Live, custom 365 domains) I eventually decided to route Microsoft hosted emails over Amazon SES instead of a VPS hosted Postfix setup. This did help a bit. The fact that all other email providers work perfectly fine and Microsoft is the problematic one I think signals an issue with their spam filtering.

Yeah, these problems are consistent with Microsoft email. I've been using Sendgrid to send emails for our SaaS and some customers who use Microsoft email don't receive transactional messages at all. There's nothing you can do about it.

Also, seen this happen with other apps too. Transactional messages don't arrive on Microsoft email addresses.

It's common knowledge that Microsoft considers everyone to be spammers until proven innocent. Getting to a point where your mails end up in the recipients' inboxes is a long path -- Microsoft doesn't care if you set everything up perfectly, as spammers can do the same -- and one has to provide them with enough data to learn to trust one's IP's/ranges/domains.

Contacting Microsoft means you have to go through a difficult dance through multiple canned replies, multiple escalation requests, and being lucky enough to get someone to actually escalate.

Silently dropping emails isn't Microsoft-specific, though. It's quite common to have a Postfix/Amavis setup set to discard spam over a certain spam score threshold.

So, basically, email lacks an ACK mechanism.

Couldn't a matrix client with email and asynchronous features be a drop-in replacement for email ? Wouldn't that be a killer product ?

SPF, DKIM, and DMARC are not to core to email protocols if I understand correctly.

There shouldn't be a way for a matrix a client to delete a message sent to another client to keep with the email usage and expectation. So matrix messages wouldn't necessarily be used as email text messages but attached documents to messages would carry email that a client would store and archive.

Email is a delay/disruption tolerant network, and each hop in the network has a positive acknowledgment that the message has been committed to stable storage (a high bar). If the final destination rejects the message a delivery failure message is sent to the originator if one was specified (it's not mandatory), otherwise the postmaster is notified.

In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.

> and each hop in the network has a positive acknowledgment that the message has been committed to stable storage

If that were true there wouldn't be silent drops in the last mile.

> In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.

ACK is not to guarantee that messages are delivered though. It's a mechanism that ultimately would let the sender know the message was most likely not delivered.

Sure.

Let me know when that new client is default installed on the phones and laptops of, say, 70% of all people. I’ll start taking a look at it and evaluate it against standard smtp/imap/pop3 then.

Sure.

But in these days and age of webmail I am not sure 70% of people have a mail client.

I don't really understand the argument of pitching mass adoption against something that doesn't exist yet.

> So, basically, email lacks an ACK mechanism.

As the article mentions, Microsoft’s server was telling the sender that it had accepted it. An ACK had taken place. But there’s no point in having an ACK if the recipient lies about it.

Tbh queued for delivery doesn't mean the email has been successfully delivered, BUT, the mail server should inform the user if a delivery cannot be performed.

Most probably this whole problem realted to silently dropping emails is intended behavior, a type of shadowbanning applied to emails to prevent spammers from knowing their email was blocked.

Too bad that this in turn causes huge issues for legitimate users

Not "should"; it's a "MUST".

   If there is a delivery failure after acceptance of a message, the
   receiver-SMTP MUST formulate and mail a notification message.  This
   notification MUST be sent using a null ("<>") reverse-path in the
   envelope.  The recipient of this notification MUST be the address
   from the envelope return path (or the Return-Path: line).

And:

   As discussed in Section 7.8 and Section 7.9 below, dropping mail
   without notification of the sender is permitted in practice.
   However, it is extremely dangerous and violates a long tradition and
   community expectations that mail is either delivered or returned. If
    silent message-dropping is misused, it could easily undermine
   confidence in the reliability of the Internet's mail systems.  So
   silent dropping of messages should be considered only in those cases
   where there is very high confidence that the messages are seriously
   fraudulent or otherwise inappropriate.

https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.4

The RFC makes clear the long-standing principle that a mail-server operator is entitled to make his own rules as to what mail he is and isn't willing to accept. But he's NOT allowed to accept it for delivery, and then silently discard it.

Hotmail was always awful for this. There are comments up-thread to the effect that Microsoft might be trying to squeeze out self-hosted mailservers. I suspect that was the case in the Hotmail days. Either it worked, and nobody self-hosts any more (FALSE) or it didn't work, and they should have given up. I suspect the same policies are still in place due to corporate inertia.

Microsoft ignoring standars? Impossible!

Fair enough.

You can't ever replace email because Microsoft, Google and all the other big players will try to push their own proprietary solution instead of agreeing on a new federated protocol.

These details about the proprietary "SmartScreen" were really useful.

We have been sending login code emails via AWS SES and have had a lot of issues with Hotmail/Outlook.com not delivering them.

My takeaways: 1) There's a risk that a shared IP system like AWS SES will get blocked by SmartScreen based on emails someone else is sending. 2) We really should look at the contents of our (trivial) transactional emails, too.

I've never hit this with GMail, but I've hit this repeatedly with outlook.com/hotmail.com accounts. There's a persistent 20% chance of any given email to my accounts being dropped silently.

It's been enough of a headache that I haven't been able to migrate off GMail and that's frustrating as hell. Especially with important medical emails..

Microsoft is also the only server operator that will randomly block me. Since I don't host any customers I have decided to simply not care about them and their customers - if someone using microsoft's email services isn't able to get email from me, it's their problem.

I can only confirm that out of all the email providers, Microsoft is the most tedious, non-standard and brutal when it comes to dropping email.

I'm coming from the perspective of a community site owner that pushes out email notifications to its members. Nothing spammy, all emails are opted-in and essential for the basic functioning of the community. They may be as simple as "person X liked your post". It's purely functional email. Plain text, no links, images, nothing. The entire site is non-commercial.

Further, the domain from which it is sent is well established (14 years old) and never used for sending out spam. It also has implemented all the best practices, like DMARC, SPF, the like.

It's not in the gray zone by any stretch of the imagination. It's as clean as it gets. Yet still, as random as the weather, it's always hotmail.com and live.com users complaining about email not arriving. It's not in their spam folder, it's just not there at all.

I never face this issue with any other email provider, only Microsoft.

There's no help at all regarding what triggers it, other than the standard document listing best practices I'm already following. They do have some tools to analyze issues, but guess what, only available for high volume email pushers.

There's also a public tool to check blocklists, some of which are shared between email providers, yet even when not blocked anywhere, you're still blocked at Microsoft.

So after all this self-analysis, I've learned to raise the ticket. Admittedly, occasionally they admit it was a false positive, and then lift the ban. But they never fix anything structurally, it's an issue that keeps coming back. It's maddening.

Since I'm in the mood of self-pity, I'll add to this Microsoft-specific problem by mentioning that it's hell to push out email in general for small websites.

For example, besides the functional email notifications, there's also an opt-in weekly newsletter. It's found deeply within user's personal settings, so those finding this setting and enabling it, surely want it.

Within the range of a few hundred to a few thousand subscribers, now you have a signal problem. A handful of users may no longer want the newsletter and do not unsubscribe (which is easily done from the bottom of each email). Instead, they may delete it, report it as spam, block the sender.

These are convenient to do from an email client, but the problem is that I never see their desire to unsubscribe. I'd gladly honor it. What happens instead is that in the case of low volume, just a handful of these negative signals may push all of your email into the spam folder.

Technically this is incorrect. It's not spam. The user opted in, the contents of the emails are not spammy, and the user never indicated to not want them anymore. But email providers don't care, it's all spam now.

To resolve this, I compare last activity on the site with the user's opt-in status, and auto unsubscribe everybody not recently active, regardless of whether they want this. This significantly reduces the chance of users flagging the email as unwanted in their email client. At the same time, now I have users complaining their newsletters aren't coming.

Bottom line: you can't win. With the best of intentions, fully wanting to comply with every technical and legal guidance in the book, it still does not work.

Does setting up MTA-STS have any effect on deliverability? I can see in my webserver logs that both Microsoft and Gmail/Google are checking the MTA-STS settings at regular intervals.

Despite all the outrage, there is also this:

"As far as I know, emails from my business custmers are branded, formatted following best practices, have unsubscribe links and a valid contact footer."

In other words, his customers are sending out the type of emails that actually require unsubscribe links. You know, _spam_. So maybe it isn't Microsoft who is the bad guy here?

Microsoft is telling this guy they received the email for delivery and then silently dropped it. If you're not going to deliver an email you either inform the sender or shutdown your mail servers and remove your MX.

> Despite all the outrage

Hah! Please, there's hardly any. Certainly nowhere close the outrage this deserves.

Oh please. It was the commercial spammers that destroyed email as a useful communication medium. Not sending all sorts of useless notifications is just another defence mechanism against those a*holes. Once in a while I still get an undeliverable notification for Chinese spam; they just configured my personal email as their return address, and some stupid mail server decided to tell me it couldn't accept the email. Great job guys, now you're spamming innocent bystanders!

So yes, telling people that their spam-load was accepted, and then just not delivering it, seems perfectly fine to me. It's a tough world out there, and it was made that way by precisely the kind of people that need "unsubscribe" links in their emails to begin with. I feel no sympathy for him.

These days communication uses web forms, Teams, Skype, Facebook, Twitter, or any other electronic mechanism that isn't email. They are all regulated by their respective owners, so they have a slightly better chance of surviving the onslaught of the advertisers. Email works fine for contact with friends and family, but not for spam, and that's how it should be.

Backscatter spam only happens with a misconfigured email service. If you are not going to deliver the mail, reply with an error during submission, don't send a automatic notification email to the return address.

Yes where I live you have to specifically opt in for email communication.

I use Outlook and the few companies who still pull this shit off end up in the spam box courtesy of machine learning. If MS is making life worse for businesses they have my support.

When these long-winded issues arise, you need to call people up. Email clearly isn't working.

You'd be surprised (or not) by the difference verbal communication makes, not only do your verbal expressions translate better, but the tone of voice may (or not) instill trust in the person on the other end.

> Email clearly isn't working.

It's working just fine for most people. Email sometimes isn't working for Microsoft customers - but that is entirely Microsoft's doing.

The general problem is by no means unique to Microsoft.

Misconfigured email servers, with spam filters and SPF/DKIM setups being the most common issues, drop emails far more often people tend to realize.

Kudos Nerd Admin epic post and work through

Hashtag MeeToo

Reply on news.ycombinator.com