Hacker News Reader

Comment by jacquesm | Hacker News Reader

I've been looking in detail at three different Supermicro motherboards but so far have not been able to spot anything. Even against a backlight there is no sign of tampering between the layers.

View on HN

The most compelling explanation I've heard is that the BMC chip could be programmed by two distinct flash chips, one for factory programming and one for some other purpose. In some SKUs, the latter isn't populated but it has a higher priority than the first chip.

Since there are many flash chips fitting the same pin out, all it took was soldering a compromised flash chip (with firmware for the BMC chip) onto pads that are already part of the design to compromise the whole system without any obvious sign that the board was tampered with (because in some SKUs, both chips were populated).

The BMCs on the newest Supermicro servers are from ASPEED. The X10 models have the AST2400 [0] and the X11 models have the AST2500 [1]. They have ARM CPUs and run, basically, an embedded Linux.

If you wanted to "backdoor" motherboards that shipped with these BMCs, wouldn't it would be much easier to just install your own "customized" version of the firmware on them? It certainly seems that it'd be much more difficult to incorporate another device into the system.

[0]: https://www.aspeedtech.com/products.php?fPath=20&rId=376

[1]: https://www.aspeedtech.com/products.php?fPath=20&rId=440

If I'm right, that's exactly what they did. When the BMC chip boots, it checks two flash chips for firmware so the attacker just uploaded their own code to one of a million standard SPI flash chips and plopped it onto the board. They didn't have to incorporate another device into the system, the system was already designed for two flash memory chips. However, to save money on some SKUs, the manufacturer left one of the positions on the board open.

Normally this wouldn't be worth talking about because most active chips are too complicated and too design/supplier specific to carry out an attack like this, but SPI flash is about as standard a footprint/protocol as you can get in EE short of transistors so if you ship a product that could be reprogrammed from unpopulated pads, you're opening yourself up to a large attack surface.

Honestly, after I read the latest BMC chip theory I was like: "Oh, shit. Have I done that?"

If possible, it is better to have separate hardware that can continuously compromise the firmware. That way your exploit continues to exist even if valid firmware is flashed directly onto the memory module.

well companies like Apple and Amazon are reflashing/updating, so that wouldn't stick.

because all it takes for it to be discovered is someone checking the SPI flash contents

By explanation do you mean theory or is it coming from somebody who has special knowledge of the situation?

I'm not trying to be adversarial, even if it's only a theory it's an interesting one, but given the amount of conflicting information we have regarding this whole mess I think it's important to be clear about what's pure speculation and what's been reported by people supposedly in the know.

I looked at this back in 2013. Here's some slides from a talk I did after spending 48 hours with them[1].

The BMC back then was by a company called ATEN, who make KVMs. The modern BMC is by ASPEED - I don't know if they're related.

What's described in the article is exactly how the old ATEN firmware worked normally. It was a spectacularly poor product from a security perspective.

[1] - http://mandalorian.com/dl/himym.pdf

Maybe you are not a high value target?

That raises an interesting question about just how targeted this kind of attack could be. At manufacture time, do the folks on the assembly line (so to speak) know who a particular board is going to? If not, they would have to add the extra chip to all outgoing boards, which means there should be plenty of them in the wild, no?

If the motherboards were customized for a particular customer, you'd know exactly who they're going to. That would eliminate the problem of letting the exploit travel too widely as well.

Right, but does that happen? I honestly don't know. Clearly a company like Amazon or Apple buys in large enough volume that they could be asking for customized MB's, but does anybody know if that actually happens? If it does, then that would definitely moot the question I was posing above...

Or swap the boards out in transit.

Seems more problematic though. You'd have to manufacture the doctored boards, extract them from the normal shipping process, keep them hidden somewhere, then swap them out for the ones destined for the target customer(s). I guess it could be done, but it seems risky.

Couldn't it be done on-demand? Apple orders X hundred boards, motherboard manufacturer makes their small modification(s) to a line that is currently producing the same models of motherboard as Apple ordered, they produce a handful, then they revert and mix in a few of those modified boards into the real order. I don't really know the exact scale, so maybe they make a few hundred / the entire order with chips in them, but economic cost isn't a big deal for things like this, so even losing money making the modified boards wouldn't be the end of the world (and presumably they get a hefty sum of money for whoever is paying them to do this).

I thought China was famous for extremely short turnarounds for industrial engineering edits, so it seems plausible that they could manufacture the boards in a reactionary way and not need to do much in the way of logistics to get them to their targets.

No comment.

If I was a high value target (and knew about it) I would definitely not let you know, if I was a high value target and did not know about it I would not be able to tell if I was or if I wasn't. So any high value target and anybody else would not be able to tell you they were a high value target.

What about the variation where you're not a high profile target and you know it?

Same here. I have four different Supermicro motherboards purchased in May for servers in my home. I'm sure there exist people and organizations in the world capable of putting malicious hardware on one of these such that I can't detect them. But insofar as I've personally examined them and the available evidence from Bloomberg, color me skeptical...

Ok now try to patch the BMC, you can actually talk to it with openipmi on local host.

I'm familiar with OpenIPMI (I use it for remote fan control a lot) but I'm not clear on what exactly you want me to try?

I really want to see someone on here with access to one of their recent boards try and report on this. I'd try it, but I sold my last Supermicro board years ago.

Try what? Updating the firmware? I do it every time a new firmware version is released.

Back around 2014-2015 supermicro had this bug that would not let you flash the main firmware. Would not happen on every machine maybe 25%. Had to derack and send a number of machines back.

Are your Ethernet shells metal, as described in the article, or plastic which the article describes as normal?

This metal shells rubbish is a key indicator that the whole story is bogus. Metal is completely normal.

You purchased servers from 2013-2015 in May? As in used servers?

You would recognize what looks like an extra resistor?

The supposed infiltrated part is a six terminal RF device. Not something that would ordinarily show up on a server motherboard. In any case, Joe Fitzpatrick has already disclosed that he used the part merely as an example and Jordan Robertson expanded that into a work of fiction.

I hadn't seen this before, but searching for "Joe Fitzpatrick Jordan Robertson" finds https://appleinsider.com/articles/18/10/08/security-research... which seems to be what you were referring to?

The original source is Joe Fitzpatrick's interview with the Risky Business infosec podcast. Apple Insider is just summarizing some of the points from that interview:

https://risky.biz/RB517_feature/

Where is the 6-terminal claim from?

The original Bloomberg article features images of a 6-terminal chip.

https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i0hZ31udMZ_...

https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i9VdsjZLS_P...

There were quite a few pictures of what is supposed to be the device in the Bloomberg article. Knowing what they say it looks like and knowing roughly where to look I'm 99.9% sure that none of the boards I have here have that device on them.

I don't have the reference handy but someone claimed to be a source and they pointed to a generic item on digikey / mouser as an example. I imagine that it got extrapolated by Bloomberg into that.

They really have no idea what they are talking about at this time and it's probably fluff.

I'm not sure why you're downvoted, except the lack of citation. Your recollection is correct, it's from the Joe Fitzpatrick interview with Risky Business, which was quoted by Apple Insider. (Fitzpatrick was named as a source in the original Bloomberg article.)

Long story short, that photo does not show the device involved.

"Robertson was unable to produce photographic evidence of the chips in question, saying they were described to him by protected sources. Indeed, Robertson in September asked Fitzpatrick what a "signal amplifier or coupler" looks like, suggesting the publication narrowed the attack package down to that particular component. Fitzpatrick sent Robertson a link to a very small signal coupler sold by Mouser Electronics. "Turns out that's the exact coupler in all the images in the story," Fitzpatrick said.

https://appleinsider.com/articles/18/10/08/security-research...

The image caption on the bloomberg story reads "Microchips found on altered motherboards in some cases looked like signal conditioning couplers". They didn't claim "that's the chip"

It has more terminals that a resistor, it's a pretty unusual package and it would stand out enough for me to spot it knowing that it is there. The area of the PCB that you could expect that thing to live in is about 5x5 cm square.

Isn't the idea that the boards weren't tampered with but manufactured by contractors including extras?

Well, that depends on your definition of tampering, but if you want to exclude manufacturing something that is not what was specced then I am fine with that but please do supply a new term.

I would definitely spot that device if it were on these boards because it was described in detail and there were some pictures of what it supposedly looked like.

A device like that is not on either side of the board and it isn't in between the outer board layers (where it would be much harder to spot, especially if the cavity would be covered by a ground plane on one side).

I am not saying it is impossible, it is just very hard to hide something like that once you know it is there. The only candidate spots left that I can not check without destruction is underneath some of the devices or inside some of the devices. That would be a different level of sophistication than the original article alluded to.

> I would definitely spot that device if it were on these boards because it was described in detail and there were some pictures of what it supposedly looked like.

In case you missed it, there is an article posted today [0] that has this quote from "Hardware security expert Joe Fitzpatrick", one of the Bloomberg sources, regarding "the supposed spy chip":

> In September when he asked me like, “Okay, hey, we think it looks like a signal amplifier or a coupler. What’s a coupler? What does it look like?” […] I sent him a link to Mouser, a catalog where you can buy a 0.006 x 0.003 inch coupler. Turns out that’s the exact coupler in all the images in the story.

[0]: https://9to5mac.com/2018/10/09/bloomberg/

Oh, that's interesting. So they basically took one guys hypothetical and turned that into a news item positively seeded with images of the hypothetical, rather than an actual device.

The original article has now dropped into the real of SF for me until they show a detailed shot of an actual board with a parasitic device on it. Until then this is a wild goose chase.

Thank you for pointing this out.

My understanding is that certain parts on the PCB were swapped out for malicous parts. If that's the case, it's probably not something that could be uncovered by a purley visual inspection. The 'spy' chips were likely made to look identical to the original parts.

That’s not what the BW/Bloomberg story claimed - it specifically called out a chip that wasn’t on the official BOM and had been added to the build.

The initial allegations from Bloomberg suggested ON the otherboard, not in, as I understand it.

There was mention of one being discovered buried inside the FR4 PCB material.

I don't think you'll find this in a board that doesn't otherwise normally have lots of other buried components ... The added cost of that extra process (using buried components) is so way higher than normal and such a board is going to look noticeably different from a normal board ... I'm tempted to think that someone told the Bloomberg guys that it was possible and the took it that it had happened

Reply on news.ycombinator.com