"Secret" Agent Exposes Azure Customers to Unauthorized Code Execution
wiz.io
wiz.io
Looks like a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.
Still not clear on what it does. Seems safe.
This codebase just smells of rot all over.
Large companies like MS often have codebases that are so ancient (e.g. HRESULTs and IF_ERR_GOTO type macros everywhere), they only get modernized very gradually if at all.
Most of the stuff I see in the wild could be considered what I call C+, basically MFC style with plenty of C love.
While Microsoft has the excuse for older code bases, stuff like WRL and C++/WinRT also has its share of not only following what is preached regarding C++ Core Guidelines love.
This is the part that gets missed when people talk about "dependency inflation" re: Rust. Absolutely, it is a problem, but most C codebases of a sufficient size and vintage are vendoring some absolutely insane hidden "dependencies" that, on average, probably get a lot less testing and attention than the average package on crates.io.
There are definitely risks with both approaches.
Honestly, they just needed to rewrite it in a safer stack. However, that still may not have saved them from all these vulnerabilities, given the scope of what they're implementing as remote management protocols. The relative scrutiny, fuzzing and manpower just hasn't been there, especially when it's obfuscated by various layers.
That is also why we now have WinRT, WinUI, Sphere OS, Azure RTOS, regardless of what Microsoft Security advises.
Whatever magic Nadella brought to revive the company is starting to wear off, at least from where I’m sitting.
How much of that is due to the new administration not liking the previous one?
> slow attrition of frustrated developers/CIOs
Citation Needed
> Moreover the torture that is Teams has got to have consequences as millions have had to suffer
Then why not use one of the 10 competitors for chat apps?
>How much of that is due to the new administration not liking the previous one?
"According to a New York Times report, after President Joe Biden took over this year, his administration examined the status of the contract and came to two conclusions– that the legal challenges could continue to stall JEDI for several years, and that the technological concept had already become outdated.
The DoD will now have a new system called the Joint Warfighter Cloud Capability (JWCC), in which both Amazon and Microsoft are expected to win contracts, and possibly more cloud players. Unlike the Trump administration, which wanted a single cloud provider, the Biden administration will be dividing the contract between multiple companies, allowing the US military to not get locked into a single vendor." [1]
It sounds as if there was a political component, but mostly that they didn't care for the deal. It was a bad look, but there is no evidence there was failure to perform on Microsoft's part.
> slow attrition of frustrated developers/CIOs That was speaking from pure anecdotal evidence on my part. A friend who ran cloud migrations for SMBs at an MSP share a huge amount of frustration with instability, incomplete services, and poor support. I've also heard from others in the cloud architecture community about concerns about products being less feature-complete, as well as concerns with Cosmos DB specifically. Personally I had some negative experiences with Azure 2015-17 but it seems like quality and UX has improved significantly since then.
>> Moreover the torture that is Teams has got to have consequences as millions have had to suffer
>Then why not use one of the 10 competitors for chat apps?
Many orgs are tied to O365 and have made the decision to centralize services around it. Adding a new app like Slack would require enterprise services to support SAML for user provisioning and new tooling for management. Most complaints I've come across center around rampant UI glitches and concerns with video conference quality relative to other options. My son was forced to use it during the pandemic and had plenty of his own complaints about it, along with those of his teachers. Here's a thread from other user reports[2].
None of this is to that that MS hasn't had wins or happy customers; any large company will have its share of blemishes. Azure's growth story has been really strong, but it seems to be slowing down and investors are taking note[3]. I'm grouchy about some issues with Windows recently and took this opportunity to compile all of the negative stuff I've been seeing about them, but that isn't to say that they are irredeemable - just not invulnerable.
1. https://indianexpress.com/article/explained/biden-government...
2. https://www.reddit.com/r/MicrosoftTeams/comments/mcx4oy/why_...
3. https://www.bloomberg.com/news/articles/2021-07-27/microsoft...
But no. Same code is on master branch atm.
What is this, a joke?
"Never attribute to humor, that which is adequately explained by incompetence"
¹ Anyone with more spare time than I want to look deeper to confirm or contradict?
Also note that neither EncryptData nor DecryptData have any way of passing a key in their API. So it's very unlikely that these implementations could be conditionally compiled in place of a real implementation (but they're the only definition of these functions in the repository anyway).
I assume this has been ported from Windows and later never implemented the ripped out components. That said, I don't know the windows API so apart from confirming that they exist in Windows docs[2] I can't assess how valid their usage was.
[1] - https://github.com/microsoft/omi/commit/edbe231042173018c529...
[2] - https://docs.microsoft.com/en-us/windows/win32/api/dpapi/nf-...
There could have been other smells involved, like the key being held in some more global scope instead of being passed in via the call stack.
I wonder what the people at Microsoft are thinking of this situation.
Still doesn't excuse having such bugs in the first place though.
Their product is pretty interesting. I did a demo a while back and their approach of building a large graph and doing a good amount of reachability (network and entitelement) leads to some useful signal. It's not entirely novel, JupiterOne has been doing something in the same vein for years now but it seems to work well. Still feels a bit rough around the edges but i thought it was interesting and could work well in situations where simple checkbox-style config analysis falls short.
They shared that query and a bunch more in this blog: https://try.jupiterone.com/my-bucket-my-data-or-is-it
security people sell security company and create a security company. color me shocked.
What rather surprises me are the comments claiming MSFT is suddenly going to go bankrupt because there is a pre-install spyware.
Have you ever worked in a Fortune 500 ?
Do you know how hard it is in those company to get anything done that has not been budgeted and planned years in advance ?
Do you seriously believe Fortune 500 using MSFT are going to suddenly migrate ALL their Azure workload somewhere else like some kind of startups run by 3 devs ?
Yes Azure VMs have spywares builtin , but last time I checked all Europeans companies that are using American Cloud fall under the « Cloud Act » legislation which « legally » requires cloud vendor to hand over ALL the data the company is currently hosting.
I’ve worked in some of the largest insurances in Europe they LOVE Microsoft and Azure , even if there is this spyware issue , they will pretend it doesn’t exist and do business as usual.
Im pretty sure everyone will forget about this news in 1 month or so.
One thing I can say, Microsoft hasn't changed a bit. It's a Win98 experience in the cloud.
Now we are fighting Microsoft silently blocking entire ASN of Airtel in Nigeria, and:
1. First, obviously pretending having no idea what are we talking about, and bullshitting us
2. Then not acknowledging. "We checked it, everything is fine"
3. When faced with traceroute - "Maybe it's a third party network provider"
4. When faced with WHOIS record - "I don't know what really to do"
5. "When faced with "this is the last day we are using you" they finally escalated it, and then "It's our DDOS team doing, we have no bearing on them"
6. So, you see it's their official policy to bullshit clients. They clearly knew something from the start, but tried every diversion, thinking they are talking to some $10 per hour anykey man.
Soon found out that Azure silently blocks a big chunk of third world ISPs, and other small datacentre providers without an option to appeal.
Maybe the best recourse here is to get rid of the spam coming out of this ASN.
North America is the biggest source of spam mail, and DDOS by far.
I recognize that bugs happen, but allowing a remote client to execute commands as root by simply removing the authorization header should have been caught by automated testing.
Patch management and vulnerability checks, right. (VM) OS configuration checks, sure, why not. "No agents or sidecars to deploy" ... hang on, how does that combine with the other two? In order to check for installed/missing patches for on-system software you have to have some kind of access to the underlying system.
Surely you are not snapshotting root volumes for your analysis needs?
I have to say the problem is not oss, not agents, but Microsoft.
By setting the provisionVMAgent property to false when creating a virtual machine, WALinuxAgent should run with all extensions disabled, and I think that's as minimal as a Linux VM can go on Azure.
This property, however, can't be set via https://github.com/ansible-collections/azure, which is of course another lovely OSS project by Microsoft. I didn't bother to send a PR.
The OMI agent seems to be a different beast that is way more obnoxious. The closest thing on GCP is probably the collectd agent and the fluentd agent installed for Stackdriver Monitoring and Stackdriver Logging? Plus whatever OS config to enable unattended upgrades.
I just learnt from this HN thread about the SSM agent on AWS. That one does seem equally obnoxious as the OMI agent.
EDIT: Looks like collectd and fluentd have been replaced by https://github.com/GoogleCloudPlatform/ops-agent
"Microsoft announces passwordless future – available across Microsoft Edge and Microsoft 365 apps": https://blogs.windows.com/windowsexperience/2021/09/15/micro...
/s, if it wasn't obvious...
How's that not a lot of people? I don't get the "only" at all.
Is this meant to be a joke?
I would have liked to see what the author thinks is a sufficient number of contributors.
I’m pretty cynical because I’ve frequently seen a lot of pot calling the kettle black accusations when the authors projects are shown to only have 1 or 2 contributors.
Intuitively, I feel it would be exactly the opposite.
Likewise, the more the code, the less the trust.
Gulp
> Yet this scenario is more common than you might think and certainly not unique to Microsoft. One of the benefits of open source is that it’s easy for developers to grab code from different projects and mix it with other open source and proprietary software. As a result, bad open source code can wind up in an enormous range of products and services – inadvertently becoming a “single point of failure.”
The fear mongering about open source in the article seems misplaced.
Any software product could lead to the same thing, and OMI was used as a software product in this instance. I don't see how it being open source changes anything and the article doesn't address this, only goes on a tangent about how "open source code can end up in any software project".
It also leaded with a bit about supply chain attacks, in a way that seemed to imply, without explicitly stating, an intentional cause to this exploit existing.
> Because customers don’t know what franken-code is running in the background of the services they use, they remain at risk and unaware.
Wouldn't this be worse with closed source software thou?
Oh, wait, fix not available in the Debian/Ubuntu official repos, only M$ trusted ones.
That's an indication that Microsoft is bad at handling security issues. Which isn't news and has very little to do with open source. Something very similar happened with Exchange, it led to plenty of people being compromised, and Exchange is not open source.
That it's open source there could an interesting argument made that it's easier for one to hunt for vulnerabilities when they can review the code. In this case what was the advantage or point of it being open sourced when it is clearly a Microsoft project for Microsoft purposes. Maybe for collaboration internally? But the open source nature of it doesn't gain much.
The open source nature–in this particular case becomes more complicated when the fix is in plain sight for a month now with a seemingly easy ability to take advantage of. So in this case closed source fix that wasn't public how to exploit maybe would have been better indeed.
The open source vs. not is a bit more of a red herring relative to the overall situation. It's certainly not the central theme, but when you overanalyze I can see how some of the analysis maybe happened.
The only recent commit was for "Enhanced security" a month ago:
https://github.com/microsoft/omi/commit/4ce2cf1cb0aa656b8eb9...
still don't understand what it does
(Sounds like it's a Linux port of some Windows management feature.)
Or, in other words: if you reaction to SOAP was “damn, this is amazing, we need more of this but with even more indirection”, you’ll feel right at home.
In fact, I’ve never met anyone who thinks that. And “all bugs are shallow” doesn’t mean there are no bugs, it just means it’s easier to find bugs.
Any company in any other field that accepted dependencies without doing quality checks to verify that those dependencies met their requirements would rightly be called grossly incompetent. This is one step further than even that as the dependencies openly guaranteed nothing and they still did not verify if it was of acceptable quality before making vast amounts of things depend on it as a critical component. This is the absolute basics of the basics and Microsoft has completely failed at even that. This is indicative of deeply-rooted gross systemic incompetence in much the same way that failing FizzBuzz demonstrates a complete lack of programming ability. As such, their opinion on matters of security and their ability to deliver security should be completely disregarded until their organization is completely overhauled.
"Currently, this project is not accepting contributions or pull requests."
Last changed Aug 17, 2016.
What's the alternative? A black-box proprietary bug that becomes an NSO or NSA exploit?
The logic on this is so bad.
Linus's law: "given enough eyeballs, all bugs are shallow".
Yea, make sure to mention Linux in relation to vulnerabilities. Then pretend Microsoft is to the rescue with OMI. A FUD piece for Microsoft masquerding as a technical article.
>“Secret” Agent Exposes Azure Customers To Unauthorized Code Execution
But the vulnerability is only for azure Linux hosts so not sure how to report it without mentioning Linux.
What’s comical though, is I wonder what they use for Windows/non-Linux hosts. Is there an OMI equivalent for those hosts? Unless it’s open source, I guess we won’t know if it has a vulnerability.