Supermicro hardware weaknesses let researchers backdoor an IBM cloud server | Hacker News Reader

Supermicro hardware weaknesses let researchers backdoor an IBM cloud server

arstechnica.com

84 pointsbauc27 comments

There's a good talk on securing bare metal cloud servers against such attacks, that was given at B-Sides in Portland, by two respected security people who worked on securing bare metal cloud infrastructure for one cloud provider: https://www.youtube.com/watch?v=PEVVRkd-wPM

Boiled down it comes to: Trust absolutely nothing. When a customer finishes with a server, wipe absolutely everything, re-flash every single bit of firmware on every single device in the machine, and don't use the standard flashing mechanisms to do so. It's worth a listen/watch.

> This indicated that the servers' BMC firmware was not re-flashed during the server reclamation process.

I'm not surprised. Unless you pay for their "enterprisey" datacenter management products (which are still relatively new), it's a PITA to perform BIOS and BMC firmware updates. Additionally, Supermicro specifically recommends that you DO NOT flash the firmware unless you are experiencing issues that a new version is suppose to fix -- unlike pretty much every other vendor (like Dell, who makes it fairly easy to do so).

Flashing the Supermicro BMC is so easy and reliable that I used to do it frequently.

Flashing the Supermicro BIOS, yeah, that's a disaster.

Could someone speak to the write endurance of BMC flash, vs whatever flash BIOS/UEFI lives in, vs NVRAM where now certain boot parameters are stored for persistence? Is it a total non-concern? It's not a user replaceable part.

It's a bit shocking that the people who develop the firmware for these embedded computers is not more concerned with their security. I understand you are not supposed to attach them to untrusted networks, but can we really call any network trusted these days?

Do people still teach about the Maginot Line in schools?

Cheap Chinese labor. Security costs time and time is money. So why bother if it isn't giving direct profits? Customers still buy them servers so, no problem in the revenue stream (yet)

>Do people still teach about the Maginot Line in schools?

No.

There are plenty of other opportunities to teach the value of making your adversary (be that adversary man made or natural) work every step of the way and they are missed too. Nobody (for large values of "nobody") gets taught about the value of layered or redundant systems until college and those that do get taught it in college usually only touch on it in their mandatory ethics elective.

FWIW I named the last firewall I configured "Little Maginot".

Another (perhaps more) common use of the term Maginot Line is as a metaphor for expensive efforts that offer a false sense of security.

It is even more shocking considering the fact that exposing the BMC directly to the internet is not only often recommended by vendor but in many cases (eg. collocation) actually only way to use it.

Jumpboxes? You need at least two for it to work though. Update A from B and then update B from A.

Do we know a one company that gone out of business due the security related problems with their products?

This highlights how the sterilization process many cloud providers (IBM in this instance) have is not cleaning out every nook and cranny.

They should, audit every bit of firmware (indeed it's odd how the researchers changed one bit in the BMC firmware and no checksum flagged it up on boot) and whilst this is daunting, it isn't that hard as they just have to compare and verify it is the same as the known safe image. Sure they could blindly reflash, but then they would miss any attempted expliotations and equally shorted the life of the hardware by increasing the odds of the flash memory failure.

Whilst people see BMC's as one avenue, a server/pc has many components, all with their own firmware and in many cases, own CPU. Be that a network card, graphics card and even keyboards and mice (though the later, not so much a factor in server environments, still a consideration).

Security is and always will be a mindset. You need to think like somebody who wants to break into your environment, and then counter those ways. But so many avenues. Imagine your sat at your desk as an administrator and one morning you get a nice shiny, cool top of the range keyboard sent, dressed up as a gift. How many would think, cool, plug it in and feel all fuzzy? How many would audit the firmware on that keyboard? How many would question the random gift at every level?

I'm sure IBM are not the only ones who would fall foul of this avenue of BMC exploitation, but I'm disappointed that for me, basic sanity checks in their sanitisation process to decommission and recommission a server are being overlooked.

Still, when you hire a car - do they audit the cars management engine firmware? Do they erase previous BT and WIFI connections stored on the radio? Well, from my experience - they don't.

Remember - you can pay an expert all the money in the World, but do check their work.

> Do they erase previous BT and WIFI connections stored on the radio?

I've pulled many contact details from cached data on rental vehicles. Always worth checking what the stuff you pair your phone with asks for and keeps.

Is there any chance they have stumbled upon the same weakness that Bloomberg tried to report[0]?

The industry at large has been pretty sceptical of Bloomberg's claims, and rightly so, but what if they just got the details wrong and it was this (or similar) vulnerability in the BMC software, rather than a dedicated spy chip, that they meant to write about?

0: https://www.bloomberg.com/news/features/2018-10-04/the-big-h...

I think it's possible Bloomberg's "tiny chip" was a BMC, and that BMC was found with a backdoor in it, but the backdoor may not have necessarily come "from factory," but was later added by someone else.

Either way, this looks terrible for SuperMicro: "Yeah, our servers don't come with built-in backdoors - it's just SuperEasy™ for attackers to add one once shipped -- please buy more now."

The article looks written to hurt supermicro here.

It looks like the blame squarely lies with IBM (for not correctly resetting the BMC between users of the machine), or Intel (for a poor design which allows this in the first place)

I'm a little curious why the BMC hardware is even exposed to the customer at all. Is it provided as a feature or is this just an oversight? Can't they just hide all of this hardware from the customer's OS?

It should sit on a physically separated network, with the firmware regularly updated.

Wasn't clear to me from the article whether the firmware was modified via the BMC's network interface or whether it was done locally over some hardware interface exposed to the customer's OS. Was this really done over the network?

Softlayer runs two separate networks, the public network, and the private network (vlans per customer, but also Softlayer tools are accessible from customer vlan, and customer vlan is accessible to Softlayer tools). The BMCs are connected to the private network. My understanding is that access to the BMC via the network is password protected, and customers don't get a password with enough access to flash via the network. Of course, older versions of the Supermicro BMC firmware had network accessible vulnerabilities, and current versions may have vulnerabilities as well. But impitool from the host can configure the BMC, and I suspect either flash the BMC or at least configure the BMC with a user that can flash it (or Supermicro tools can likely be run on the host to flash the BMC)

Bare metal customers get access to the whole machine. We get to access the BMC for remote management (console, power management, etc). Apparently, you can configure the BMC via tools running in the host OS, without needing to authenticate to the BMC; I've used this to fix BMC provisioning errors, that would otherwise require a SL tech to fix via physical console in the bios screen.

Ah, interesting. I wrongly assumed the SoftLayer bare metal servers might be closer to EC2's bare metal offerings where, from the outside, you're given the same set of management tools as a VM, and from the inside, a fairly limited hardware surface area is exposed to you.

Does this vindicate Bloomberg or is it a different BMC vulnerability?

Totally different. This is a compromise via a known firmware issue with the standard components on a motherboard.

The Bloomberg article was about a hidden chip being installed on a motherboard that provided a backdoor.

Strikes me as odd that they didn't acknowledge the recent history of reporting on Supermicro BMC issues (in order to clearly distinguish them).

If you read about both, it becomes pretty clear (at least, it did to me) that they aren't related.

Bloomberg's story was about a (supposed) backdoored chip installed into the server at the factory, this is about flipping a bit in the firmware installed on the existing BMC.

Nope, nothing to do with the Bloomberg story. While we're at it, Bloomberg still hasn't issued a retraction or provided more evidence to support their claims. This is despite the massive number of refutations from the entire industry.

Reply on news.ycombinator.com