Supermicro server motherboards can be infected with unremovable malware
arstechnica.com
arstechnica.com
Then you've already lost.
The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access.
> The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
This is good practice but it shouldn’t excuse poor security at the hardware level.
Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
Yes, all of which can be reversed by another admin in the future. That is expected.
It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
If you have admin on windows you can flash the bios on regular motherboards with firmware that refuses to change.
The vendors even sell this as downgrade prevention!
More generally, when you get down to the bottom of the pile of elephants, you are requesting some software currently running on your computer to write some bits to some kind of storage medium.
But there is no law of physics that says the software must to do as you ask! If the software is malicious, it can refuse. It could even pretend that it updated the bits but not actually do so.
"Oh, but I booted into $OTHER_PROGRAM and it writes the bits!"
Maybe. But how do you know that the boot loader faithfully loaded it? You don't. Maybe the boot loader is malicious and patches your firmware updater so that it won't actually write new firmware.
If you squint and tilt your head, it kinda looks like Ken Thompson's "Reflections on Trusting Trust".
What chip are you using to bit bang? Is that chip directly or indirectly controlled by the firmware? Usually it is.
It makes it easy to brick a system.
True in the common case, but this can/should be guarded against by disk encryption and secured boot chains.
It's really the "permanently" which is the design flaw. Boards should have a mechanism to recover from bad firmware, and the same mechanism is useful to recover from a bad flash.
Make the flash chip removable, or leave a JTAG. Or have a bit of actual ROM with the write lines not even connected and just enough of a firmware to be able to reflash the main one.
This is exactly the kind of barrier you want for something with so much power over the system, otherwise you're not much better off than where you started as physical access allows for quick swaps of chips.
Meanwhile it provides no meaningful resistance against physical access because someone with physical access can swap the entire board or a dozen other things.
It’s not common in modern IT, and the only time I do it myself is in the course of preserving vintage hardware
And if you need to desolder to remove the malicious code, it's pretty reasonable to call it unremovable.
You might see that as a facetious comparison. But the number of orgs which actually would desolder the chips in that circumstance is very close to the number which actually would scrap and replace. And if 99% of orgs won't actually do it when needed, then a "works in theory" method of re-securing servers is real-world useless.
If administrator access is equivalent to ownership, then I strongly disagree.
Flashing data? Fine.
Permanent? Not so much.
You don't go ahead and erase the same disk once a week. Decommissioning isn't something that occurs for the same project, once a month.
Its not the same operational process.
Should every software config require buying new hardware because the initial config gets permanently flashed with an e-fuse to only allow a single write? You could even make a security argument for such a setup, but good luck getting approval for your 15th motherboard this quarter because you typo'd the config.
Also, dban and degaussing is not entirely equivalent -- from a practical perspective the equivalent is hard drive shredding (because the hardware cannot be used again in the old/non-malware config -- dban and degaussing are more like factory default resets). Do some organisations need to do this? Sure. Should we design systems with the assumption that any mistake means that the hardware is destined for the shredder? I would hope not...
If you are magnetically destroying hard drives as part of decommissioning, that's not really the same thing. You're not using admin access to do it, and you're not making a change that permanently applies to all future use of the device (because there is no future use).
If you do it right, the data erasure.
Anyone can delete a file. Nobody wants to ban deleting files.
Even if you changed the default from "failover" in the firmware you're just one CMOS reset (for whatever reason) away from it reverting back to putting the BMC on whatever network interface. This should really be something you can jumper off.
Is this documented?
If it's documented, it's not an excuse for not reading documentation. You're not supposed to throw stuff (hardware or software) in production without reading the documentation.
So you'd definitely have to have it connected to the internet somehow, even if very indirectly, and in an independent manner (different network with no direct routes).
It is common to keep admin and backup functions on separate network interfaces, on a disconnected network. You have to physically connect to the network in a secure location to use it.
There are others here more qualified to comment, but I do have some old rack mount enterprise servers making a racket in my basement, so I can provide some sort of half-informed opinion here--
Besides the security issue with the BMC here, it sounds like SuperMicro _also_ has absolutely insane defaults.
Every system I have has a separate, dedicated NIC for the BMC. There is an _option_ which is disabled by default to instead have it share one of the other interfaces. So the only ways you're going to "accidentally" put the BMC on an insecure network are:
1. Rebooting the server, going into the BIOS configuration, going into the options for the BMC, and explicitly telling it which other network device to use.
2. Physically accessing the server, attaching an ethernet cable to a clearly labelled BMC port, and attaching the other end to an insecure network.
From what I'm reading here, SuperMicro's default approach is apparently more "eh, just use whatever happens to be plugged in at the time". So even if you do have it running on a separate, secure interface... if someone unplugs it, it's going to switch to using whatever other connection is available!
To connect to the BMC on my servers you first need to be on the internal network already. Then you connect via wireguard to the router on the rack. Then you can connect to it and given a username/password you can log in. I would be pretty pissed if a cable got unplugged and that meant that the server decided to instead throw the BMC _on the interface connected directly to the public friggin' internet_. And this is just equipment I use for play and hosting my own random junk!
How do you track the chain of custody of your servers? Do you sample them at random to ensure they aren't compromised?
Bloomberg never backed away from their story about Chinese implants in Supermicro servers. Perhaps this is why?
> infect it with unremovable backdoor
> stop paying server so company rents server to somebody else
> ????
> profit!
It's not impossible to do in the field, but you can't really count on vendors surfacing that interface usually.
Pogo pins are only really needed for mass production, especially for reducing repetitive stress injuries. For one-off updates, if a header isn't populated, it's easy to hold an unsoldered header in place, for long enough to flash an update.
But AFAIK, tangible evidence never surfaced. [1]
--
0: https://news.ycombinator.com/item?id=18176620
1: https://news.ycombinator.com/item?id=19910537Supermicro is one of the only vendors that tries to prevent this attack at all through RoT.
Other vendors you can flash whatever unsigned firmware you want. It’s very useful for adding in microcode for intel engineering samples, or malware…
Look up Intel pfr.
At least for 4677 Intel stuff, gigabyte & HP and others let you modify the firmware and flash it.
If it were me I'd assume the majority of BMC firmware out there from all vendors: 1. Is full of many many exploitable vulnerabilities 2. To the extent they patch holes it will be whack-a-mole because the economics do not permit large investments in software quality. 3. Many server owners will never install a patch anyway.
Unfortunately, Supermicro doesn't use it yet for most of their servers. Probably because they sell an extremely expansive license for their own software so you can use the Redfish API.
Dedicated KVM devices?
If you build own servers that's an option to consider but most off-the-shelf servers are sold with BMC (so you pay for it even if don't want it). May be some low end brands sell servers without BMC but if you are looking for relatively reliable hardware you'll likely get a server with BMC.
That should tell you everything you need to know about the security risks involved.
In other words, it’s a neat feature, but maybe not one many customers actually request.
If you remove that component from their value prop, they're not that much different from Dell.
All of it is a far cry from the offerings of Dell/HPE/Supermicro, which rely on others to provide the software that turns the hardware into real infrastructure.
[0] https://oxide.computer/blog/systems-software-in-the-large
For what it's worth the last time I had to procure a small cluster the offer Dell made was ridiculously overpriced, didn't meet the specifications I asked for, and the whole experience didn't inspire confidence in the software they were pitching either. We were a small underfunded startup so we were never gonna drop 200k on that storage solution (0.5PB HDD + ~80TB SSD), but if we did have that kind of budget we probably would have gone with one of the smaller but more focused parties there instead like maybe TrueNAS, I bet there's a beautiful market for Oxide as well in that segment.
Most current BMC platforms are older than seL4
Most run on hardware that is not supported by seL4, or at least on hardware where it has not been validated.
Not to mention that a task manager would be needed as well as tons of other services which aren't provided out of the box, and don't share the verification provided guarantees.
Some motherboards just have a physical jumper that prevents BIOS flashing. This happens infrequently enough as to warrant it for one server, or 10 servers, or maybe 100 servers. Likely unpractical for 1000 servers though.
Make this one simple enough and add an EPROM for it. Effectively a security chip for the oob. Extra points for secure enclave-like verified boot.
> Often we see.. great security.. compromised by other great ideas for mgmt and other things.. starts to weaken its security posture.. want to keep Caliptra very clean [via OSS firmware transparency]
https://www.servethehome.com/the-ocp-dc-scm-hff-is-taking-ov...
These days it’s common to do firmware updates to address small issues or even support the new CPU that was launched after the motherboard was manufactured.
I could see manufacturers adding a write-protect physical switch for those who want it. Make it opt-in and toggleable.
Using the method you talk about would mean that this kind of update wouldnt be possible, 99% of users would never toggle with a switch to update firmware.
This would be a huge burden in the server world too, to unrack flip switch, update, revert switch re-install.
I assume you mean specifically motherboard firmware updates, because firmware updates are actually pretty common, for most server grade motherboards vendors ship updates about every other month[1].
1. https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...
I think these days, the stub "BIOS flashback" is the trendy thing, where you can plug a flash drive into a magic slot and press a button to flash without even having a CPU installed.
This offered the same "brick-resistance" feature with the added benefit that people weren't stuck if they tried to pair an old-stock mainboard with a new CPU that wasn't supported by the original firmware release.
TBH, I'd rather they go the complete opposite direction: replace the soldered EPROM with a SD slot and a $1 MCU that reads the card and emulates a ROM chip. That could be configurable to write-protect the card, or you could just trivially swap it if you didn't trust the firmware image for any reason, while avoiding the fumbliness of modern tiny 8-pin flash chips. You could socket a big old-fashioned DIP ROM, but will people feel comfortable even trying to pry that out of a $10,000 server even with the appropriate chip puller tool?
Regular desktop motherboards can be flashed from windows.
Yawnnnn.
What this article is describing is something far more likely— a firmware attack that doesn't require specialized hardware.
“In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm. “These two companies were subsequently involved in the government investigation, where they used advanced hardware forensics on the actual tampered Supermicro boards to validate the existence of the added malicious chips.”I'm not sure where you got that idea. The article describes a tiny microcontroller, attached to the read pins of the BMC's boot flash, flipping a few bits in transit from the flash ROM to the BMC SoC as the BMC boots. This is not only practically possible, it's very similar to the technique used to hack the original Xbox and by many console mod chips. And is sufficient to boot the BMC in a vulnerable state for the next chain of an attack.
Nothing about the exploit claimed in the article was impossible or even novel.
That said, I'm not aware of any physical boards found to have the compromised hardware outside of those Bloomberg claim to have witnessed.
Yes, in theory it is possible to prevent these kinds of infections without resorting to secure boot (e.g., by insisting that all the suppliers of components of the motherboard start designing components that cannot be pwned) but so far all the computers you have actually been able to buy that are immune to these kinds of infections achieve that immunity with secure-boot technology.
It seems to me that, in this situation, secure boot’s only role is to provide a false sense of security, which could make recovery from the attack less likely.
In contrast, verified boot might somewhat mitigate the damage from being able to use the BMC to write arbitrary data to the SPI flash chip. Emphasis on might — at best I expect that it would require an attacker to be a bit more creative in how they design their exploit payload.
And I think this would deliver a slight level of protection from the BMC: tampering with the firmware image or key enrollment / secure boot state _should_ both break the UEFI root of trust and alter the PCR state and break everything downstream. Of course, all UEFI implementations are holier than Swiss cheese and there are probably a lot of ways to use the BMC to dump or modify memory post-boot anyway, but it does do something.
And Secure Boot is implemented in, and configured by, the firmware that the BMC can overwrite at its whim while entirely bypassing all the fancy CPU-hardware and SMM protections that are supposed to prevent writing arbitrary data to it.
To the extent that a mechanism not controlled by firmware will detect such an attack and extend PCRs accordingly before executing a single instruction from the compromised flash chip, it might partially mitigate attacks. But that part isn’t Secure Boot.
Secure boot can include the hash of the firmware, computed by the root-of-trust that can't be tampered with by this attack. So the exploit will make the keys stored in the TPM inaccessible.
This will make the tampering conspicuous, at least.
The BMC usually has full access to system memory as well, so if you can get the timing right, you could replace the secure boot verified image with your own after verification.
Also, re: BusinessWeek, hey look a hardware backdoor installed on servers. Pretty sure IPMI vulnerability fits the bill for most of what was described.