Hacker News Reader

Comment by ThinkBeat | Hacker News Reader

This whole article seems a bit odd to me. What is "the product" ?

Presumably this is not related to earlier problems with SolarWinds.

Did MS screw up. Yes.

However, all things have bugs.

I takes one person finding one bug and exploiting it. and there are enormous resources going into finding one, and I am certain that this is the only one.

I am sure the NSA is sitting on a pile of them.

Whereas the developers have to think about everything that can happen and protect against it.

Does this make Microsoft different from its competitors?

I think Microsofts strategy is somewhat similar to Linus:

Where security patches are often not part of new releases due to the burden of establishing what the consequences of bigger changes would be, and the fact that security people dont do sane things.

(But you can of course pull them and make it part of an in-house distro.

https://lkml.iu.edu/hypermail/linux/kernel/1711.2/01357.html

View on HN

> Harris said he pleaded with the company for several years to address the flaw in the product, a ProPublica investigation has found. But at every turn, Microsoft dismissed his warnings, telling him they would work on a long-term alternative — leaving cloud services around the globe vulnerable to attack in the meantime.

That is not a screw-up, that is a deliberate decision.

You might want to read the actual article.

My understanding is that it was a two-part exploit:

1) The Solarwinds product was hacked to allow backdoor access to organizations' on-prem networks.

2) The hackers then took advantage of the "Golden SAML" vulnerability in Microsoft's Active Directory Federation Service (AD FS) to leapfrog via "seamless SSO" from the on-prem network into the organization's cloud resources hosted by Microsoft.

The article is all about how various Microsoft leaders and staff did not fix #2, because many said it would never be an actual issue exposed to the world.

This is extra damning because Microsoft is selling components at the core of both governments' on-prem and cloud systems, so if they don't take security extra seriously, their systems can present passive vulnerabilities.

> You might want to read the actual article.

ProPublica articles in general are structured in a way that makes them a pita to extract actual useful information from.

It's in the article's headline.

And at the risk of annoying everyone, a GPT summary:

This article investigates how Microsoft, in pursuit of profit and market dominance, overlooked significant security vulnerabilities that left the U.S. government and other entities exposed to cyberattacks by Russian hackers. The whistleblower, Andrew Harris, a former Microsoft cybersecurity specialist, discovered a serious flaw in a Microsoft application used for cloud-based program access. Despite Harris's persistent warnings over several years, Microsoft delayed addressing the flaw, prioritizing business interests, particularly securing a lucrative deal with the federal government for cloud computing services.

The security loophole was within Active Directory Federation Services (AD FS), which if exploited, would allow attackers to impersonate legitimate users and access sensitive data without detection. Microsoft's decision to deprioritize this issue, despite internal and external warnings, eventually led to the significant SolarWinds cyberattack, affecting numerous federal agencies and demonstrating the consequences of the security oversight.

Microsoft's response to these accusations has been to emphasize its commitment to security, stating that they take all security issues seriously and review them thoroughly. However, ProPublica’s investigation reveals a culture within Microsoft that sometimes places business growth and competitiveness over immediate security concerns, reflecting broader issues within the tech industry related to balancing profit-making with customer security.

The article sheds light on internal conflicts, the company's handling of security vulnerabilities, and the broader implications of such practices for national security and customer trust. It also highlights the challenges faced by whistleblowers and cybersecurity professionals in advocating for swift action on security issues within large corporations driven by profit motives and competitive pressures.

Microsoft had a known, high consequence, security flaw that they did not acknowledge or fix, they had evidence that indicated it had already been exploited and they knew they had limited to no ability monitor for exploitation. This choice lead directly to the SolarWinds hack that happened in 2019 was discovered in late 2020 and acknowledged by the USG in early 2021.

Many companies make bad choices around security for profit, however that factors I listed above make this extremely egregious.

I would seriously question any use of Microsoft products in any security conscious organization after this reveal. I also hope that anyone negatively effected by the Solar Winds sue Microsoft for knowing about the vulnerability for years without fixing it or disclosing it.

> However, all things have bugs.

There are bugs and there are critical flaws you’ve been warned about. This is the latter.

The fact that this was known by Microsoft but not fixed is the story.

Because as far as I can tell, there was no "vulnerability" here, it's just how the product works. Stealing an OAuth key is just as bad. Stealing a domain's krbtgt key is just as bad.

Businesses want that when they login to a computer, they are SSO'ed in to all their apps. That's how ADFS works, you authenticate to it using kerberos and it issues you a SAML token. Here they stole apparently the key used to sign the SAML token so they could generate their own.

Unless there was some vulnerability that exposed the key publically, I fail to see how in this particular incident its Microsoft's fault.

>Stealing an OAuth key is just as bad

What is an "OAuth key"? Do you mean an OAuth token? No, Golden SAML is worse than stealing an OAuth token, because an OAuth token is valid for 1 user, but Golden SAML can be used to impersonate any user. Also, OAuth tokens expire, but Golden SAML doesn't expire (although if you steal an OAuth refresh token, that won't expire).

>I fail to see how in this particular incident its Microsoft's fault.

Andrew Harris wanted to warn customers about the weakness, and tell them they can prevent the weakness by disabling seamless SSO. Other Microsoft people said no, that would alert hackers to the attack, we want to keep the attack secret, and it also would jeopardize our contracts by making the default setting sound insecure. Then Golden SAML was published publicly, so that first reason was no longer valid, but Microsoft still wouldn't tell customers they could prevent the attack by disabling seamless SSO. Then Solarwinds happened, and Microsoft finally advised customers to disable seamless SSO.

I think there is too much confusion in the details of the actual attack.

You have to steal the private key for the SAML signing certificate for an app. The correct answer would be to scope any token to only have access to what the app has access to, the second layer which is documented in their 2020 article, is to require mfa on admin actions, and the 3rd layer is to disconnect azure admin accounts from on-prem admin accounts preventing this type of attack.

But disabling SSO altogether is non-starter for most businesses, what are we going to do tomorrow? Spend months recreating 100,000x accounts in various applications, no.

We decrypt ssl traffic in our company, someone steals the private key and now can read the entire stream including your bank account details, lets stop decrypting ssl traffic because someone might leak the key? The answer from the infosec communinity has been its worth the risk.

If all those other solutions are better, why does the article say this:

>In the immediate aftermath of the attack, Microsoft advised customers of Microsoft 365 to disable seamless SSO in AD FS and similar products — the solution that Harris proposed three years earlier.

And did Microsoft advise those other solutions prior to Solarwinds happening?

This is the mentioned article: https://techcommunity.microsoft.com/t5/microsoft-entra-blog/...

What it says is "be careful when using federated trust relationships, because if one of your trusted environments is pwned, it will be trusted by the others". That's very obvious.

And about "disable seamless SSO", I only found this: "On-premises SSO systems: Deprecate any on-premises federation and Web Access Management infrastructure and configure applications to use Azure AD." (Seems pretty basic too, especially considering how vulnerable on-prem ADs are).

The original article seems to paint this MS page as a security advisory or vulnerability notification, while it just seems to me to be a very very basic security guideline.

I think those things the article is advising are the same things Andrew Harris wanted to advise customers to do 3 years prior, but Microsoft didn't want to, because it would make the default configuration sound insecure (it kind of was), jeopardizing government contracts, especially since various government systems would break if those config changes were made.

I get what you're saying, but from my point of view, this seems like something that doesn't need to be advised, because it is so trivial. Yes, if someone pwns my AD, then they can also pwn my cloud if i'm using some sort of federated trust. Even if i'm not, and both systems are completely separate, they just need to steal passwords from the cloud admin, which should be easy given they're already domain admins.

Maybe Andrew being overly cautious, was assuming most government users didn't know these basic facts, and should be warned anyway? Was MS pushing back on his report because communicating something like this to users would probably sow too much confusion?

That would still a failure on MS's part, but would make for a much more boring story. The article makes it seem like Andrew discovered an atomic bomb and MS pushed it under the rug. The reality seems much more bland.

But still, could you elaborate on the default configuration being insecure? I know next to nothing about Azure/Entra, maybe I'm missing something important.

It is pretty boring. Where I would blame Microsoft, there needs to be an easier way to setup AD, AAD, ADFS, without having a bunch of people be domain and global admins, like out of the boxed roles and better gui. Every ad deployment I’ve ever worked in is insecure due to complexity of secure deployment. So people running it are going to be logging in domain admin /ga to do basic crap like add a new hire.

>this seems like something that doesn't need to be advised, because it is so trivial

According to the article, that's not the reason Microsoft gave for not advising it. The reasons they gave were (1) it would make governments scared and jeopardize contracts and (2) it would let hackers know about the attack.

Also according to the article, the NYPD weren't aware of the problem until Harris warned them of it, then they quickly disabled seamless SSO:

>On a visit to the NYPD, Harris told a top IT official, Matthew Fraser, about the AD FS weakness and recommended disabling seamless SSO. Fraser was in disbelief at the severity of the issue, Harris recalled, and he agreed to disable seamless SSO.

>In an interview, Fraser confirmed the meeting.

>“This was identified as one of those areas that was prime, ripe,” Fraser said of the SAML weakness. “From there, we figured out what’s the best path to insulate and secure.”

>But still, could you elaborate on the default configuration being insecure? I know next to nothing about Azure/Entra, maybe I'm missing something important.

I'm not very familiar with Azure either. I'm getting most of this from the article. It sounds like the weakness is that by default trust federation to Microsoft 365 is enabled. Microsoft's post-Solarwinds article recommends disabling it.

> "disabling seamless SSO"

It is never going to happen in the corporate. Never.

The article says Andrew Harris worked with the NYPD to disable it for their setup.

And Microsoft themselves advised customers to disable it after Solarwinds.

> What is an "OAuth key"? Do you mean an OAuth token? No, Golden SAML is worse than stealing an OAuth token, because an OAuth token is valid for 1 user, but Golden SAML can be used to impersonate any user. Also, OAuth tokens expire, but Golden SAML doesn't expire (although if you steal an OAuth refresh token, that won't expire).

Stealing the OAuth token signing key, since then any fake OAuth tokens signed by it would be considered authentic.

There isn't necessarily an OAuth signing key. The OAuth tokens might not be signed. They might be random values, which act like a password, with a hash of them stored in a database so they can't even be stolen from the database.

Even if they are signed, it doesn't need to be as bad as Golden SAML, because OAuth tokens have a short expiration, so the signing key can have frequent automatic rotation, so any stolen signing key will quickly be useless. For the refresh tokens, they don't have fast expiration, so frequent rotation won't work, but you could have a hybrid system where the OAuth tokens use a frequently rotated signing key, but the refresh tokens are random values with hashes stored in a database.

This is ignoring security in depth, weaknesses, and security architecture. When ignoring that, you can not pretend, and MS did pretend, that you had a good enough stance on security. Fixing discovered vulns alone is mandated, it gives you maybe half a point, but the other 9.5 points or at least 5 before you can claim you care about security require more than fixing known vulns or waiting for world scale incident to "respond". You have to prevent issues.

It is true that nothing is 100% secure. Sitting on a major security vulnerability internally with a motivated employee pushing to fix it and doing nothing for business reasons is not negligence, but malice. People in the chain of command need to be held accountable for this.

>What is "the product" ?

Human attention sink where you can throw ads and other propaganda, what else?

Reply on news.ycombinator.com