Hacker News Reader

Comment by lovelearning | Hacker News Reader

It was reported that the security auditor used during Elemental's acquisition detected this compromise. I assume they found it in a randomly selected board. Either they were very lucky, or hundreds of boards were compromised.

Now, I think all motherboard manufacturers - and especially high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. Under what circumstances is it possible that SM's QC missed this out on so many boards? Won't it affect things like the power budget, weight, and latency? What do professional EE people here think?

View on HN

    > ... high end server manufacturers like SM - use sophisticated automated tests and quality control on boards.

MFG test guy here (not supermicro!)

Not necessarily. Automated production tests are there to configure and exercise the system and confirm it works as specified. Such tests are good at things like finding bad solder joints, pick-and-place mishaps, misconfiguration of firmware and weeding out product that fails functional test. That, by itself, is hard enough.

Such tests are not _really_ able to detect things which no one is expecting, or worse, things which an adversary has specifically designed to avoid detection. Sure, if something gets "discovered" during a failure analysis, a test or process can be created to specifically address THAT problem in the future.

To find "unknown unknowns", you need an audit of some kind and that is definitely different from production test.

Thank you for the insight. I was rather naively imagining that every board is subject to some kind of x-ray image matching with the original PCB design to find differences. Not that simple, I see.

Some manufacturers do X-ray boards.

Typically this is done to check that BGA devices (https://en.wikipedia.org/wiki/Ball_grid_array) are soldered properly. Usually not done on every board, but only if there's a problem suspected. When it happens they tend to focus only on particular BGA components or suspect copper traces rather than the whole PCB.

Usually you scan boards at a percentage determined by statistics (SPC can get complex). However the checks are automated, so you would have to perform the checks in a different trust zone (eg: different country).

I'm guessing the answer is obviously yes that this type of x-ray would easily be able to discover / distinguish something the size of what's been described? (1mm x 2mm from what I've seen thus far?)

The x-ray picture would contain information sufficient to detect that thing, but it's quite plausible that the process/procedure of analyzing that x-ray would not find it. If they're looking for extra hardware, they're going to detect it, but if they're looking for bad solder joints, they're going to detect bad solder joints but not extra harware.

Right. It doesn't exactly align with what a normal auditor would be looking out for, I suppose. Thanks, I appreciate it.

I've only seen X-ray as a destructive test. AOI is pretty commonplace on a per-unit base, but whether it could or could not detect some modification depends on lots of details we don't have.

It's not uncommon at all to do AOI on all or a percentage of the PCBs. X-ray is more useful to identify solder issues or defects in the raw PCB itself.

Exactly. I've never seen x-ray used post SMT. We've used it to post-mortemly debug BGA issues.

They would be able to detect rogue component(s) if AOI, Automated Optical Inspection, was done.

However, if the source files already had the component(s) on it, it would pass all tests and QC.

In other words, most automated tests are regression tests.

Depends on where it was added. Doing it as part of the assembly process would require having access to the pick and place software along with being able to load reels of the implant into the machine. A well placed insider may be able to make that happen, and automated optical inspections would be fed by the modified pick and place program, so checks would pass as the new component is part of the build.

Thanks for that info. I assumed that once the boards came back to SM's US premises, a second level of optical and other tests are done by SM employees. But perhaps that's not - or was not - their actual workflow.

Would be pretty wasteful to wait until the hardware was shipped to the US to look for manufacturing defects.

EE here, looking at the size of the chip, the weight and power difference is going to be too small to measure. Latency will probably not be affected when the chip is dormant

If it replaced a DNP part, it could have been found at a flying probe test. Half the expected impedance on a few of those pins?

Also AOI might have picked up a difference.

Flying probe testing is unlikely to be used for this type of product.

Sounds like a randomized sample of boards needs tested against a reference then.

AOI would miss the chip if it were buried between layers or look exactly like a part it was replaced for.

I wonder how the auditing team detected it. X-rays?

They might have noted an anomaly in the SMC given the acknowledged SMC firmware exploits (maybe a beta test run?) and investigated the SPI lines to the SMC as a response.

Are you basing this off the photo in the Bloomberg article that shows a small (0402?) SMT package with three terminals?

I wonder if that's an actual photo of the chip in question.

FTA: https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iWRg56pmx50...

> [...] a picture of the alleged implant. This shows a 6-pin silicon chip inside a roughly 1mm x 2mm ceramic package – as often used for capacitors and other so-called ‘passive’ components, which are typically overlooked.

It might be possible to mount an attack from a board location occupied by a legitimate passive component (or a single SOT-23 transistor, something like that). In that case the attacker would only need to replace a reel of legit passives in the PnP machine. Visually they'd be identical.

SOT-23 only has three pins, seems like it would be challenging only having one data pin.

The alleged component in question had six pins, if it were passive it must have been a bank or array of passives since individual passives generally only have two pins each.

Based on the captions in the original story, I think it's just a photo of a random signal conditioning coupler, which the chip in question was said to strongly resemble.

But I'm not sure.

One of the bloomberg articles claimed that in some cases it was "thin enough that they’d been embedded between the layers of fiberglass" -- like as if it were a passive in a blind/buried via?

This seems like a very sophisticated attack.

Not a sophisticated attack, but a standard industry practice for high value, high density boards. I first saw a buried passive 5 years ago.

Sorry, I didn't mean to suggest that the attack was with a passive. Clearly, this package is shown to have logic. But it was meant to look as if it were a passive. In some cases surface mount, but in truly devious ones it's much better hidden.

Well my own idea was along the lines - if they can burry passives, burying a small IC should also be possible

SD cards have the raw IC die buried in the PCB. If you crack open a cheap SanDisk card, you will see that it is a marvel of engineering. Instinctively one thinks of advanced manufacturing as being reserved for high end goods, but in reality it is used widely in cheap mass produced things to bring the cost down.

Passives don't have enough pins to do much of anything, unless it's an array of passives in a single package that happens to be on power, gnd and data lines at once.

But, I'm not sure but it seems like a bit of a red herring. Reading the article the chip in question doesn't sound passive at all: "The Supermicro board here appears to have a QSPI chip, but also a space for an SPI chip as a manufacturing-time option. The alleged implant is mounted in part of the space where the SPI chip would go."

edit: so maybe it was an array of passives that were meant to go across every pin of this protocol in a single package.

Did you see the comment above that QSPI boots up as SPI and then switches over to QSPI?

It can't be QSPI, he's forgetting the clock and (potentially) chip-select lines.

That’s my main takeaway from all of this- Elemental is great at their job and didn’t just do a half assed attempt at an audit. They actually did what they were hired to do. I wonder how rare that is.

You perhaps meant not Elemental, but Elemental's auditors is great at their job.

> ... In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says. Nested on the servers’ motherboards, the testers found a tiny microchip ...

Oh yes, my mistake yes, I thought Elemental was the name of the third party security company. Do we know the name of that company? They need some major kudos for this.

There used to be a Nortel Division up there that had an electro scanning microscope for this sort of thing - I imagine they´re still there under a different name.

> what circumstances is it possible that SM's QC missed this out

Rogue insider, paid or patriotic, likely both.

> affect things like the power budget, weight, and latency

All three would have no measurable change, falling into measurement error margin. Its a death sentence for Supermicro, nothing will help to restore trust. They should publish a detailed post-mortem analysis, though.

> They should publish a detailed post-mortem analysis, though.

More of an obituary at this point.

They took pains to say that this was happening with sub-contractors in China building motherboards in excess of what Taiwan could handle. So to build trust they can pull back on that extra manufacturing capacity and move it elsewhere maybe South Korea.

> So to build trust they can

You can't put the toothpaste back in the tube

Yeah, SM has the most to lose here. Regardless of whether the compromise is true or non-existent, they really should consider becoming more transparent about their QC, I think.

The Bloomberg article says suspicions were raised during a routine audit that resulted in a deeper audit.

The routine audit could have uncovered hackers in Elemental's corporate network doing things like ensure certain machines go to certain customers, etc.

Reply on news.ycombinator.com