Microsoft is failing to deliver emails that mention newclimate.org | Hacker News Reader

Microsoft is failing to deliver emails that mention newclimate.org

newclimate.org

177 pointsmritzmann80 comments

Interesting. I just sent myself a test mail from and to an O365 account which included the URL of the article.

As the article claims, it doesn’t get delivered. Not even in the spam folder.

Other email between the same addresses goes through without problems.

EDIT: the Exchange admin center has the email quarantined with reason “malware”, discovered through “URL detonation reputation”.

Confirmed this on sending a link to the article to myself and didn't receive it.

O365 Hosted, but non-governmental.

Same behavior here. Sent 4 test mails, 2 from and 2 to an O365 enterprise account. Both directions containing a different URL were delivered. Both directions containing the URL of this article were not delivered (though the from O365 did get put into sent items folder).

I wonder if its some sort of spam detection tool being aggressive?

Microsoft is infamous for exactly this behavior among folks who self-host their MTAs. At least with Gmail you can usually figure out where the false positive is and try to do something about it via postmaster tools. Once Hotmail blackholes you, though, that's it forever as far as I could ever tell.

This wasn't the only reason why I switched to Fastmail after 17 years of selfhosting, but it was among them.

Used to work a lot with email, and found all email providers to be workable for false positives (not necessarily "great", but workable/acceptable). Except Outlook 365. Even Google/gmail is tons better at handling these types of things, and that's saying a lot.

A lot of large scale email providers are making it more and more difficult to run your own email infrastructure without also owning/controlling the IP range in which your sending servers are issued their IPs.

I understand the logic, but most of us don’t have control of the IP allocations in which our email servers sit.

Proofpoint was the one that got me to move to Fastmail. They had filtered the entire IP allocation in which my mail server sat at a major cloud provider. Their stance was I could either use whatever email service that provider had, demonstrate control over the entire IP range (just to de-list one IP), or move my server to another IP that wasn’t filtered.

I understand the logic to an extent, but those of us who have enjoyed running our own email services are sort of left in no-man’s-land.

Getting support for something like this at Microsoft is impossible. They don't care at all. They will reply with standard responses from some supporter without any access or knowledge of the inner working of the system.

I do understand the ideer of letting a large IT organisation handle you email. Working email is a high priority, if not Microsoft or Google can make it work, who can?

But as soon as there are any problems a little out of the ordinary, its clear that choosing something like Microsoft or Google is a really bad idear. It's not possible communicate with anyone who know anything.

As more and more are moving their solutions to Microsoft and Google, they will be in a position to strangle any smaller providers.

They are a thread to the free internet!

"Have you tried SCF /scannow?"

"make sure all windows updates are installed"

"Update your drivers"

"Have you tried a clean install?"

If you are O365 customer, open support ticket and I am pretty sure they have 48H SLA to respond or reach out to you.

"Idea", by the way, since you randomly misspelled it twice, I supply the correction in case you're a very proficient English learner.

So uh. Well, I'm from the american south and I've heard plenty of people say idear instead of idea. Mostly older people or people with very strong rural accents/dialect.

Apparently also a british thing I just found out: https://english.stackexchange.com/a/32630

That it is pronounced differently in some regions of the same country does not change its spelling within that country's borders.

There exist spelling variations for various words within a country's borders. Why would this one be any different?

https://en.wiktionary.org/wiki/idear

Yeah, I mean if we all understood regardless, it can’t matter that much in an informal setting.

Assuming (as is most likely) that this is some kind of unintentional behavior of a spam filter, it really highlights the risk of depending on these few, huge centralized services for email.

Yeah, it's true that having all email in a few very large baskets is risky. However, running an email server is also not without risk and requires significant IT investment at any kind of reasonable scale. Keeping up with all of the auth mechanisms, spam, block lists, security vulnerabilities etc takes real focus. It's not something an IT shop with a just single guy should take up...

I think the real risk is relying on email at all anymore. The underlaying protocol dates back to the era of "let's see if we can get bits to move at all" which predates any kind of "how do we know what we are getting is authentic" style of design. There are plenty of other avenues for communication that don't succumb to the many inherent pitfalls of email.

I am honestly confused what big risks you believe to be connected to running your own email server, and what you think would be a significant IT investment?

For context, I have been running a Debian mailserver (postfix + spamassassin + dovecot) with autoupdates - and occasional major version updates - for family and friends since 2007. Barring the occasional period of being preferentially delivered to the spam folder, I have not experienced any problems. My major benefit: I am sure that if a mail is sent to me, I will receive it.

The system is running on one of the cheapest root servers from Hetzners used server market, and it also runs an odd set of other websites and VMs, so the IT investment is limited. I also consider the administration and update tasks as a form of continuing education in my profession.

Do you distill your own alcohol as well?

I wonder if those who say they're "honestly confused" why someone would let someone else do undifferentiated heavy lifting for them are not really confused.

Well, the parent comment was not saying „it’s too much work“, but „it’s risky“!

I can totally understand why someone would not do it because it’s too much work.

For family and friends fine.

When you have 30,000 users worldwide is it daunting.

I agree! Therefore I’d suggest more people should host for family and friends :-)

Except, family and friends isn't the target here. It's large-ish, professional organizations with a lot more surface area and visibility than your family email server. I've run this kind of friends/family setup, but I've also run a small org setup with a lot more visibility. It's definitely two different beasts... and that was a decade ago.

It depends what. I self host a lot of services for family and friends - but ony the ones I have complete control over.

If there is a problem with something, I want to have a hope of fixing it (either someone already had the issue, or I can open an issue, or I can try try to fix the code/configuration myself). These are great.

Mail not so much. This can become a continuous struggle to get off blacklists, manage spam, troubleshoot deliveries, ... All these are not dependent on me and if I get into a blacklist or if Google stops to accept my mail I am cooked. This is the reason that after self-hosting email for a few years I got back to having it done by people whose job it is .

> There are plenty of other avenues for communication that don't succumb to the many inherent pitfalls of email

The pitfall in this very article is the fact that communication was centralized on a giant near-monopoly which imposes it's arbitrary rules on users by filtering whatever they want.

So in that context, can you tell what these other avenues of communication are that don't suffer from this exact same problem? I'm guessing you're thinking of various 100% proprietary channels, all of which suffer from the problem of being centralized and users and content can get arbitrarily banned or blocked for no reason.

At least with email you can simply stop using microsoft-hosted email and move elsewhere and your problems go away while still remaining email accessible to everyone.

> I'm guessing you're thinking of various 100% proprietary channels, all of which suffer from the problem of being centralized and users and content can get arbitrarily banned or blocked for no reason.

You are looking for an open+federated protocol. Those definitely exist.

For example, an older one that I'm well aware has probably long-ago-peaked is XMPP. One can host their own XMPP domain and talk to other XMPP domains just like email can. It also has the ability to send content, presence, pub-sub notifications, offline messages (much like an email message) ... you name it. Storing contact lists and associations combined with the fact that messages come from authenticated sources by default means that spam is a lot harder to accomplish (or at least a lot easier to squash). All of this and XMPP is a 25 year-old protocol; This protocol can legally drink in the US. Just imagine what else exists today! :)

However, the problem is not what technically exists. The problem is what is popular. Email is still extremely popular and ingrained in so many places. It's easy to extend and put up a quick webpage that submits an email from some user-input. It's easy to send notifications/advertisements to. Attachment protocols have largely been sussed out. It's even used quite commonly as a second form of authentication. You can expect most people to have an email address while a much smaller percentage of people expect to have any other single-form of electronic communication.

> You are looking for an open+federated protocol. Those definitely exist.

Yes I am. Like you know, email.

That's why email is so awesome. I can easily spell out my email address to anyone in the world and they can now send me a message. We don't have to share platforms or apps or OSs or providers or anything at all, other than access to a TCP/IP pipe. And I will then receive that message. No company can interfere with that.

Yup, xmpp checks all of those boxes too. The addresses even look the same.

> Yup, xmpp checks all of those boxes too.

Well, except one.

Go find a random nontechnical person and tell them "send me an xmpp" and see if you ever get that message!

Email not only has all the technical advantages noted above, but also the advantage of universality. Just about everyone who has ever used the internet, no matter how nontechnical, will have sent emails so they'll know how to reach you.

I covered that too. “The problem is what is popular.”

> There are plenty of other avenues for communication that don't succumb to the many inherent pitfalls of email.

But few, if any, that have the many advantages of email. Different tools for different needs and all that.

No need for the conditional. If this was unintentional, it highlights the risk of depending on a few hugely centralized services. If it was intentional it really highlights the risk of depending on a few hugely centralized services.

Sure, but it hides the risks associated with the same kind of problem (assuming unintentional) in a highly distributed system. This is especially when the problem is distributed over many fiefdoms.

Being able to call out a single giant entity on HN is much easier than trying to get the world to agree on fixing something. Source: we still use SMTP and we still have no common way to authenticate a sender without a centralized email system. (Yes, I know about SPF, DKIM, etc.)

I have an O365 account in one of Microsoft's Government clouds. I sent an email from my personal (privately run) email to my o365 account with only newclimate.org in the body of the message. The message was sent to Quarantine in the Gov cloud - where it shows as Malware. Microsoft shows "URL detonation reputation, Mixed analysis detection". Seems Microsoft thinks newclimate.org is hosting malware.

I sent a second message from my personal account to my O365 account, with just my company's URL in the body. This one was delivered right to the Inbox.

> Microsoft thinks newclimate.org is hosting malware

Maybe they are? Not saying intentionally, but perhaps they have been compromised?

I was trying not to make a judgement call and just report my findings.

In my experience with Microsoft's URL detonation, it could go either way and be a false positive or be real. In one case where I had a definite false positive, opening a ticket with Microsoft resolved the issue within a few hours. Both myself and the entity with the false positive are government cloud customers, maybe our experience would be different in the commercial cloud. Interestingly, this issue seems to affect anyone using Microsoft hosted email without regard to which cloud you are using. Different data centers, separate implementation, but some shared data apparently.

Yes outlook does some very poor spam filtering. Case in point, my food delivery email arrive hours later on Outlook, while the same arrive on gmail instantaneously.

This is going to be very interesting to see the mechanics of this, since Microsoft claims newclimate.org is not on any blacklist….yet an individual organization can appeal to Microsoft to have the url unblocked and apparently it makes it work, but only for their organization.

That seems on its face like an impossible contradiction to me.

Interesting to me that this also blocks sending the IPCC report even as an attachment.

This certainly seems oddly targeted, but I doubt that Microsoft is intentionally to blame. More likely their infrastructure is compromised and someone is selling blacklisting as as service lol.

Or there is an out of control ML algorithm here or...

> This is going to be very interesting to see the mechanics of this, since Microsoft claims newclimate.org is not on any blacklist….yet an individual organization can appeal to Microsoft to have the url unblocked and apparently it makes it work, but only for their organization. > That seems on its face like an impossible contradiction to me.

I could see some sort of whitelist that overrides some sort of machine learning based blocking.

Not to say it's super reasonable but that's the only reasonable thing I could think of, with perhaps not applying it wholesale being that it would allow spammers to just include a link in background color to bypass spam protection or something

Do you checked this? https://www.bleepingcomputer.com/news/microsoft/microsoft-di...

I have been of the personal opinion that email should be end to end encrypted. This would be another reason why. Think about how much personal information is easily exposed by non-encrypted email. It would be like having the 99.9% of the web sites still using http as opposed https. I have seen more than one doctors office have an agreement that they want you to sign so that they can send one unencrypted email. (Most people don't pay much attention to the agreements that they sign and non-technical users likely don't know that their emails can easily be read by several parties.)

I don't understand why so many of the comments seem inclined to assume this is unintentional. Given the extreme sensitivity of the issue and the extreme amounts of money/power at risk / in play on all sides, it seems highly likely that this is intentional (although not necessarily official MS policy - it could of course also be internal bribery-fueled sabotage and/or a hack).

Hanlon’s razor. Occam’s razor.

I had the exact same thought. However, we’re it intentional, I’d also assume more climate sites would be affected. Still, the thought that MS could be on the payroll of an oil company? Seems banal to me.

I think it's much more likely to be accidental but its easy to imagine that if someone had a beef with this particular organisation they could easily falsely report many of their mails as spam/malware. After a certain number of unrelated reports the filter is likely automated.

I tested this just now from my gmail to my work office365 and received it just fine, fwiw.

Dare I ask what is the "climate change community" is? Lobbyists?

> Hundreds of governmental and non-governmental organisations working on climate change appear to be experiencing disruption to email communication when their communications contain any reference to the NewClimate URL.

Anyone emailing something that contains their URL.

This includes the IPCC report, which is fairly significant if you're a climate researcher:

> No organisation using Microsoft email services can currently send the IPCC Sixth Assessment Report of Working Group 3 as an attachment to anyone else (newclimate dot org URL appears 11 times in the report). The same applies to hundreds of other relevant scientific papers and reports from any organisations, where NewClimate URLs appear on the reference lists.

They don't seem to be climate change denier kooks, which by the name of the org (NewClimate) might lead one to fear.

Individuals and organizations involved in climate change communication, investigation, or any other aspect of interacting with information regarding climate change.

This is exactly why organizations should run their own email servers...

In general you should avoid doing your email through giant providers like Microsoft and Google. They just don't care. Smaller providers are more interested in ensuring that everything is actually working.

This is generally not practical for medium to large companies.

So true. Also running your own mail server can be a nightmare with bigger problems.

Both approaches have their downsides.

The small providers can get caught in the gears of large providers or even smaller providers using idiotic RBLs (sometimes a single one that causes a permanent reject). Nobody dares block Gmail or O365 however.

But you won't have an undiagnosable spam filter. Depends on your risks.

Unfortunately those kind of decisions tend to be left up to C-level individuals who generally only look at the costs associated with that decision.

Hell, even MIT has succumbed to this pressure - they are in the process of migrating the entire campus from an onsite hosted Microsoft exchange server system to cloud hosted microsoft 365 email system. To the laments of the users and the IT staff who have to support it.

From a system that has served a massive user group like MIT successfully with little downtime for over a decade to a system that has already caused multiple issues - even when they are still migrating people after 6 months...

Not just cost, they are also allergic to things like mailbox quotas and maximum attachment sizes. Everybody wants to shoot 25+MB attachments at each other and store it in their mailboxes and then summon it from a search 5 years later. Microsoft has recognized this customer demand and it's something m365 does pretty well. Your onprem PST gets corrupted if you sneeze at it but microsoft will let you have a 100GB mailbox with 1.5TB archive on their service and you'll never lose anything or suffer a slowdown.

> ... but microsoft will let you have a 100GB mailbox with 'unlimited archive' on their service and you'll never lose anything or suffer a slowdown.

Seems unlikely that will actually be the case if even plain emails are being silently dropped without notice just by have a certain url in them.

Much better to have it on premises, and actually get notified when emails are mysteriously dropped.

oh my bad. I'll turn the spam filter off for your address. let me know if there's anything else I can do and be sure to rate this interaction 5 stars.

I actually do think this is an ATP issue. I've had a few premier cases for it and the guys manning the desk at microsoft dont know how to deal with it. We always have to get a japanese speaker to raise the case in that langauge because the JP support guys are way better than the english.

The problem is that no decision maker at Microsoft is actually going to promise in writing that even 99.9999% of the email data will be retained, and does not somehow mysteriously vanish, no matter how much money you spend with them. Let alone an actual 100%.

So it's pointless to make such a claim that no one at Microsoft HQ even approximately believes.

Exchange on premises is dead. Out rather dying a slow death in a legacy environment.

Have you tried to set up MFA for an on prem Exchange system? Well, it is simply impossible. MFA on activesync ? Impossible.

When you have to support such a legacy environment you are better off moving to Microsoft 360 (I think this is their new name), or gmail or others similar players.

It's also somewhere the C-level can point a finger (besides at himself) when something goes wrong.

There are a host of problems with this. Record retention, searches, legal holds, space management.

O365 just handles all of this out of the box.

We ran our own some time ago and worked closely with the state government who also hosted their own email at the time. They had Barracuda e-mail filters that they used, and they would constantly flag our emails to them as spam at random times. Of course, we could just pay Barracuda a verification fee and get the green light. They couldn't even whitelist us themselves! But those fucking spam filter appliances were everywhere at the time. It sounds like this is just another case of a shitty spam filter.

Then a bunch of the emails you send will end up not being delivered to Microsoft and Google.

Not very subtle. I’m sure it will be chocked up to “a misfiring spam filter” but after John Stewart’s unceremonious firing from Apple for his truth seeking content plans one begins to wonder.

To be blunt, I don't understand how people who work in technology, and who often see the result of bugs when dealing with an enormous amount of complexity, are so quick to jump to conspiracy theories when the "misfiring spam filter" is infinitely more probable.

To emphasize, I think New Climate's blog post does correctly highlight the fact that when "the algorithms" go haywire, it feels like it takes multiple acts of God to get a human to actually fix the issue. But I'm just tired of the lazy appeal to "What The ELiTes DoN't WanT You TO KnOw!!!" whenever a bug pops up (and, ironically, who the "elites" are solely depends on what tribe you identify with - in this case, if you're pro-CO2 reduction, the elites are evil corporate masters seeking more use of fossil fuels for their profits, and if you're on the other side the elites want to use climate change as an excuse to stifle the economy and keep the peons in their place).

The elite are the group of people you can't question or criticize in front of some normal person without destroying your reputation.

No group should be protected from being the subject of open conversation in an open society.

Yeah I don't think it's a conspiracy. I think it's a bug, or they are trying out some "AI" on their spam filter and it needs better training.

But it also illustrates what could happen. Microsoft or Google could decide to silence email on certain topics, or they could conceivably be ordered to do so. Would they refuse? Doubtful. And they are so big and handle such a large percentage of email that it would have a real impact.

This is likely what is going on. But!

Information control is becoming mainstreamed and it’s hilarious that people are so willing to assign to incompetence or error what has been clearly demonstrated to be systems of control and squelching of difficult opinions, especially from big tech.

You see “big tech” (Microsoft, Google, etc.) has been responsible for so much crushing of “misinformation” lately I think it’s comical that people give the benefit of the doubt when these things happen.

Shouldn’t we at least be mocking them for their hilariously dangerous errors and sometimes failures? Isn’t it honestly time to stop calling it a “conspiracy” like it’s not happening.

I’ve run large scale DC operations. I know these things happen. But I seriously think it’s time we started questioning information control practices when they pop up like this — it’s not conspiracy when you have dozens of blatant examples.

I had intended it as just a random thought because it struck me as funny. But seeing the backlash is anything but heartwarming and I really think people should seriously consider what could happen and is likely going to happen and defend against it happening.

There is so much hostility to the idea that big tech would do this, when in fact they have been regularly doing it. Why in the world would you think this isn’t possible and how could anyone honestly assign “accidental” blackouts of organizations to conspiracy in 2023?

the bad PR and people assuming the worse is the price the company pays for choosing to have no human support at all. And the price isn't even high enough to discourage them, it makes no sense to complain about people holding the companies to account.

Microsoft blocks billions of emails a day. I'm sure these are just getting hit in some cross-fire instead of nefarious scheming.

It’s always true until it isn’t.

> Stewart informed staff Thursday that he and Apple executives agreed to part ways, according to multiple show staff members with whom Stewart discussed the matter. He told staff that the company had concerns about the subject matter Stewart planned for three shows during the upcoming season. Those topics included China, Israel and artificial intelligence.

https://www.cnn.com/2023/10/20/media/jon-stewart-apple/index...

Some of the other reports about this only mention two of those three. Make of that what you will.

https://www.businessinsider.com/jon-stewart-show-apple-tv-en...

The hell does this have to do with on prem vs. cloud hosted emails for organizations?

Read the comment I responded to, then whine at him instead of me.

Well I appreciated it.

Reply on news.ycombinator.com