Microsoft Azure silently install management agents with vulns on your Linux VMs
twitter.com
twitter.com
Experienced, smart, and savvy programmers will make all sorts of mistakes in their code, even stupid mistakes. It takes an immense amount of effort and savvy to reduce, mitigate, and recover from bugs in production code. Meanwhile, management applies pressure to ship new features. The kinds of teams and people that manage to ship high-quality code on a regular basis are teams with a lot of different types of people on them—you need someone who can make the case to management that these efforts need resources and are in the best interests of the company, you need someone to drive team culture and figure out what kind of practices will reduce bugs, you need people who work on automated tooling, you need people to run disaster scenarios, you need technical coders who can make frameworks that are easy to use but hard to misuse.
Nobody I met came with any of those skills out of college.
Multiply the difficulty when you’re working with distributed systems, like this one.
Exactly. Experience is key.
> Meanwhile, management applies pressure to ship new features.
This quantity over quality mindset, combined with the industry's rampant ageism and veneration of newness over all else, is making things worse at an alarming rate.
Reason why I've been uninstalling the agent on each Azure VM since 5 years: you can't make mistakes in code you don't have, at the cost of losing integration with the dashboard.
This wouldn't be an assignment in an entry level uni course.
"Write an agent running on a machine capable of providing remote command execution, authentication and that must report OS metrics externally". Next week's lab : "Recursion".
I don't remember management-agents pre-installed in other cloud providers I've used over past several years. I've not used Azure instances much, Only their server-less services and it seems like Azure & Oracle are on same boat as far as management-services baked by default in the images(of course vulnerabilities have not surfaced for the latter ...yet).
But Oracle seems to be supporting Bring Your Own Image (BYOI), which I assume those who are serious about security are already using.
Is that accurate? Is this some kind of joke?
Open ports to webservers like Apache,nginx etc. aren't affected by this issue.
Like, the basic flow would check the validity and, implicitly, the presence of the auth header. To bypass auth in the case of the absence of the header itself would need to be an explicit conditional. IF no header, then authenticated. Right? That’s crazy.
I suppose I could look at the code.
I'd probably forget to write it... but it would be useful and easy.
They could have supported cloud-init but went for the usual not invented here approach which was a shit show.
If you want first class Linux support look elsewhere. Anywhere else!!!
"Microsoft announces passwordless future – available across Microsoft Edge and Microsoft 365 apps": https://blogs.windows.com/windowsexperience/2021/09/15/micro...
I'm sure it's fine for someone who does IT work for a medium/large business, but for an independent user the UX just plain sucks compared to any number of VPS providers.
Whenever I have to work with Azure I get uneasy... (SAML...)
This is not great. In terms of ethical concerns I would rather use Azure than AWS for example.
https://developer.microsoft.com/en-us/microsoft-edge/tools/v...
Most of these tools are janky, poorly tested and I’m sure contain dozens of vulnerabilities of their own.
Edit: also, it’s invariable one dude with poor hygiene working on the agent at these companies. He’s usually at the back, in the closet. Most of the engineering work is on he backend and reporting, so the agent gets no peer review or formal security review.
Spoiler: we'll still be finding this stuff in 2040
You do lose some functionality, though.
By default, SSM Agent is preinstalled on instances created from the following Amazon Machine Images (AMIs):
Amazon Linux
Amazon Linux 2
Amazon Linux 2 ECS-Optimized Base AMIs
macOS 10.14.x (Mojave) and 10.15.x (Catalina)
Ubuntu Server 16.04, 18.04, and 20.04
Windows Server 2008-2012 R2 AMIs published in November 2016 or later
Windows Server 2016 and 2019
Of course, if the agent's verification of who it's talking to is as good as in the case of Azure, all bets are off.
---
[0] I've just checked this on an Ubuntu EC2 instance. The SSM agent is running, but it doesn't listen on any interface. No custom configuration was done it.
The software vulnerability can be exploited by anyone with network access to the machine.
If you remove the authentication header, that check never fires, and it considers you authenticated. Then it proceeds to let you run any command.
Now the point is, anyone who can send you messages can strip the authentication header, so anyone who can send you messages can execute arbitrary commands.
The GCP agent similarly does not listen to the network.
Both the Google and AWS agents are written in Go, too, so they're unlikely to have the classic C/C++ errors, more likely to use libraries rather than reinventing the wheel, and using a higher-level language often makes the logic easier to understand. Neither of those are foolproof or prevent logic errors, of course, but I would still expect a lower bug density all other things being equal.
I love programming in go, but I disagree with this point. The golang library ecosystem is absolutely less mature compared to C++.
Rust and Go are a pleasure to write in, but they don’t magically fix every problem and frequently CREATE problems because they’re still under development. In this case, the missing auth header vuln has nothing to do with the underlying language.
Two things here :
- This vulnerability is a logic error (no header => root), not a buffer overflow. It could have happen in any language.
- C and C++ don't play in the same band anymore. Most security vulnerabilities affecting C generally do not affect C++ (no stack based string handling, no VLA, no void* everywhere, proper RAII, proper type safety)
And yes for developing a minimum-memory-footprint system daemon in 2021, I would use C++ or Rust but definitively not C.
I hope that should do it...
It's worth mentioning many of these features which deploy OMI do so in a way that does not set up a listening port. Therefore, the impact of the vulnerability is pretty low. Indeed, I just spun up a Ubuntu VM and while it got a vulnerable version of OMI, it doesn't have an open port.
The LA table VMBoundPort will let you assess if you've got one configured for listening on the OMI ports. I think a lot of the people in this thread, and possibly the people writing the articles aren't Azure SME's, hence all the panic and grumbling and eye rolling.
Also I don't care about the monitoring and analytics, so there's that