Supermicro boards were so bug ridden, why would hackers ever need implants?
arstechnica.com
arstechnica.com
> > The rogue instructions, Bloomberg reported, caused the BMCs to download malicious code from attacker-controlled computers and have it executed by the server’s operating system.
It's using the fact that the BMC has unfettered access to the rest of the machine to compromise the code running on the server itself. That's valuable even if the BMC itself is on a private network inaccessible to the attacker.
There's your problem.
It’s a pretty common configuration.
There is no such practical network which remains such a network for long. All networks must be assumed to be byzantine as they certainly will be compromised at some point, if they're not already.
I know of at least 2 places where this is still the case (that or a remote IP KVM...).
The BMC should be on a trusted network, but most likely isn't.
After it left the factory. But can't the firmware be installed by someone while the board is in the testing phase?
A valid conclusion might indeed be to stop them being accessible from the outside, by installing such server boards in a network-topologically secure location such as a landfill.
This isn't a fair assessment of the situation. Networks that aren't entirely trusted and controlled can cause spectacular failures. By knowing this, administrators can use BMCs safely. My Poweredge server even came with a warning sticker that had to be removed before the DRAC port could be used.
In general, tools can have "pointy parts" with which the user could harm themselves so long as the risks and proper uses are documented and explained adequately.
Ummm, because if you need your hack to be reliable, you can't rely on someone else's bugs to be there when you need them. You never know when they'll be fixed, or just replaced by new bugs.
A long time ago when setting up computers and networks was driver version hell, we had a short list of manufacturers' computers that we'd do setup included in the price instead of on-the-clock. This came about when a shipment of about 20 Dell computers, all supposedly of the exact same model# and revision, required about about 11 different setups, because the various chips on the board were different. They were clearly just using the chip-of-the-week>from whatever supplier was cheapest -- great for their price points, but every variant required a different driver for some subsystem. So the list was created and Dell was not on it (it was IBM, Compaq, HP, DEC, to show when this was).
That's solved now by hiding it with the much more automated OS and networking setups, but it is easy to see how the Chinese spies would be in the same situation -- some buggy boards are wonderfully exploitable, but how do you tell that the version going to your target wasn't changed by some revision that wasn't even noted in the Rev- listings? Better to insert your own bug if you want to actually get the job done.
If there is anything working against the Bloomberg story it is that it is too plausible. Often reality clashes with imagination, but the Bloomberg story contains almost everything you could imagine happening.
How much has the US spent on the F-35? How much has China spent on making artificial islands? Yet engineering a chip and bribing/threatening a few factory workers is beyond the pale?
It is much easier to compromise firmware directly or modify ICs that are already part of the design. The risk of being caught is much lower and it would be stupid to attempt anything more elaborate.
This requires [et c.]
Think that's impossible? Not at the nation-state level. Not in communist countries where everything is property of the government by default. Not in capitalist countries like the United States, where entire nuclear facilities are replicated in secret. [0]
[0] https://www.businessinsider.com/the-us-built-a-secret-replic...
As a bonus, subverting the BMC firmware is much harder to trace to the source since it could be injected by in so many ways by so many different people.
Why use a thermonuclear device when a hand grenade accomplishes the goal?
If anything I think the idea that a Chinese manufacturer with complete access to the hardware having to execute some exploit towards the web interface to get access is far fetched. So is that you could pretend to update the firmware (surely no one is going to notice that the new version doesn't have the features you wanted?) and that dumping the firmware would be inconvenient (it would be the first thing you did if you suspected something).
All the Chinese government has to do is go to the factory and tell them "flash the BMC firmware with this image" where the image is subverted (but operationally indistinguishable) BMC firmware. It doesn't get much more straight forward than that.
NSA demand backdoor on CPUs. other States figure out how the backdoor works and how access to it is allowed on the silicon. Instead of attacking ever changing firmware and whatnot, just develop something that will work on that authentication component of the always-present backdoor. The backdoor interface won't change so often as it is dictated by the NSA and likely designed by a committee.
Done. Now the economies of scale allow you to just place that one component, which will work all over the place, for a very low price/complexity (all you really have to do is to place it in the input signal for the CPU and all it have to do is to filter a very specific pattern. the rest is just visual and camouflage).
This also gives you the benefit of not having to work a payload for your attack depending on capabilities. You will always have the same capabilities. It makes perfect sense. And makes it extremely cheap!
Often reality follows [somebody's] imagination - i mean you have those think tanks where people sit and imagine things, and the sponsoring agencies like CIA/Pentagon/NSA or their foreign equivalents take many of that and implement. Many people everywhere had the thought of full remote control of the computers - Intel implemented it as Intel ME feature of CPU because Intel controls CPU. China controls motherboards, so they did on the motherboards.
If all of these unnamed sources are unnamed because they were adversarial members impersonating government officials, then that would make a little more sense why current government bodies are not just staying mum, but actually denying knowledge of the story.
With the software attacks being much more feasible as the Ars article points out than a hardware attack, then it would also make it so that the vehement denials from affected companies would be true as well. The whole thing could be a large disinformation campaign to strike at the very core of what many would otherwise consider reasonable security.
A better tinfoil hat theory is that the whole story was fabricated by Russia, to (you know, as always) sow chaos.
So in the planted story hypothesis, there is certainty of negative outcomes with only the potential for positive outcomes, and those only intangible, while in the this-is-real hypothesis, there is near certainty of some tangible benefit with good probability of significant tangible benefit, with only a potential, distant, deniable risk of negative outcome.
In my experience verifying a source means weeding out that possibility before publishing... e.g, cross-checking data from a third party (background checks, employment history, social media accounts, public records), then photos of credentials, video chats, etc. Then you cross-reference information with other sources on the story, etc... conspiracy is possible, but unless Bloomberg is inflating the number of sources it has, it would have to be a massive undertaking (state-sponsored).
Anonymous doesn't typically mean someone just calls up and says something and then it's off to the presses. They know exactly who gave them the information, but they're protecting the identities.
Maybe claims of "fake news" would be a lot less common if more people knew what went into verifying information before a major news outlet publishes a story.
We know this very state does it to laptops brought into the country by corporate execs (https://www.securityinfowatch.com/blog/10861870/keeping-secr...) but again, there's no way they'd do it on a factory line?
I don't get it. Are we so confident that Amazon, Google, and Apple wouldn't fall for this that we refuse to believe it? I know everyone is saying "show us a compromised board!" but it's very likely that the our Gov't would ask that either (a) those boards be left in place or put in a honeypot so the enemy doesn't know that we know or (b) get handed over to them for forensics, etc and probably destroyed.
For the most part in my nerd circle of friends I've noticed that the only ones that believe the Bloomberg story are the ones that were or currently are in the intelligence community. Everyone else thinks it's Bloomberg being dumb because of that whole "they pay journalists based on how they change stock prices" article.
I just hear skepticism based on lack of actual evidence, as there has been, to date, exactly zero. For a hardware back that could only have been done at a large scale.
Then you come to find out it's actually happening. It just seems like such a huge thing that's hard to comprehend. I, personally believe it's entirely plausible.
BTW, Amazon doesn't know anything about security. Every day I observe examples of people who work there, wittingly or unwittingly, doing things to erode any security that might happen to be in place. It's almost entirely run by below average people scrapped up and recruited from the dregs of third world countries.
But unnamed sources are known to the reporters and as "senior national security officials" they should be easy to verify and difficult to fake.
My guess is it's a subgroup of one of the agencies running a relatively independent operation to boost distrust of China. A rather inexperienced or at least incompetent group, based on how awkwardly it's gone over.
(Not that I've come to any conclusions... I think there's more info to come on this.)
Imho The Register also points out some interesting details about this whole thing [2]
It's not really that surprising, fits perfectly into Trump's narrative of "They took our manufacturing, it's time to take it back to the US!". Gotta start somewhere, telling everybody China is selling a lot of bad apples seems like a simple enough start.
[0] https://s3.amazonaws.com/static.militarytimes.com/assets/eo-...
[1] https://web.archive.org/web/20170721190725/http://www.sepio....
[2] https://www.theregister.co.uk/2018/10/04/supermicro_bloomber...
Once discovered, such an attack would be burned for every affected board as people would replace them.
But this article also points out a case where, even after SuperMicro had published a patch to a serious BMC firmware vulnerability, 32,000 servers in the wild had not been updated a year later.
So, if software updates aren't always speedily/reliably deployed in the wild by customers, can we really expect hardware to be speedily replaced?
Considering the US went to the trouble of wiring the North Atlantic for sound to catch Russian submarines during the cold war, and tapped undersea cables using divers and submarines, this is so implausible for a nation state? Large state actors specialize in activities for national defense that make "no sense—from a capability, cost, complexity, reliability, repudiability[sic] perspective".
A deep-pocketed attacker isn't going to risk flashing the firmware with a non-oem one on a brand new board leaving the factory. That probably gets quality inspected somehow later on anyway whereas a visual inspection is just a rubber stamp (IE: OK if the box isn't crushed or wet).
Not to mention a customer in the field who experiences problems is likely to report their firmware version to Supermicro support, whose poking around could expose the entire project.
There was an article recently about how hardware is "magic" and the IT world mostly takes it for granted. Putting an extra chip on the board but making it completely transparent to software debugging techniques is the best way to go. The board is almost certainly going to be flashed at least once and probably audited several times in it's lifespan by IT, but the hardware is never going to get more than some compressed air blown on it. Nobody repairs these things at the component level on a scale that matches how frequently firmware gets flashed or checked out.
Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that....
The solution to these problems is to put critical code and critical secrets on discrete, simple SoCs where you actually have a chance of defending both hardware and software attacks. Apple and Amazon understand this because they already do it. The difficulty is building your software systems (firmware, kernel, etc) to make use of these secure elements, not to mention making them available for ad hoc application software. It's an extremely difficult integration problem, and even when you succeed you haven't.
For example, AFAIU Amazon's servers have secure elements to perform attestation of the box; it's utilized by their hypervisors to authenticate VMs for things like KMS. But it can't actually protect the data in the VM itself, such as the secrets obtained by dint of the attestation. It can't even prevent taking control of the hypervisor. All it does is help Amazon define a fixed security parameter--that you can't impersonate their hardware nodes on the network. That's extremely useful, but ultimately extremely limited.
> Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that....
Why not? X-raying every board, inspecting every single component, making sure it matches up with the documented specs and perhaps with a proven-good board... if you're replacing a component with a different one, or adding a component that wasn't there before (which is the case in this alleged attack), even if the component looks harmless and even if it's tiny, it can still be revealed by a detailed comparison of the board against specs. A component that's the size of a grain of rice is still a component that can be detected.
I'd go further to say it isn't just about the accuracy of the bloomberg piece, but implies bad things about their journalistic integrity. I mean, get real, Ars doesn't have an investigative journalism team. The one-sidedness of the bloomberg article becomes very apparent.
If you want to just disable a very large number of machines to create economic damage or cripple infrastructure, a hardware implant would do just fine. And you wouldn't need to be very careful as to where it ends - if you make enough of them, they'll be everywhere.
If 1% of all MacBooks have a similar backdoor, there are about a dozen at my building.
Even if you do directly connect to the flash module and directly write to it through SPI, if the attack is being loaded by an additional module between the flash memory and the BMC, it could still inject additional data into the BMC's boot. If you're not physically listening to the SPI data being transmitted or knew what to look for in the final environment of the BMC, you wouldn't know it had happened.
Maybe it takes away from the firmware hacking version of the story because now folks are looking at components as being the source of hacks and not the firmware on the components, leading to a false sense of security when they invest mightily in analyzing components with X-rays? I could see that outcome as being plausible. If the ultimate outcome is simply to change corporate priority towards futile component verification and away from firmware verification then indeed the firmware verification vector remains safe for the attacker.
Or, "Extra chips on the motherboard? You're about a month behind the news cycle and didn't you hear it was all BS anyway"
But the false sense of security interpretation is plausible too.
I wonder who holds conferences on the cutting edge research of manipulating media
I understand what they mean, but that sentence still hurts to read.