What Happened with Supermicro?
hackaday.com
hackaday.com
but, my reasoning? my reasoning was that if it was a widespread physical hack, someone would discover it fairly quickly; I mean, people were tearing these boards apart, looking for anything. There's no way they could keep that quiet, even if they were willing to disappear people.
I can tell you that within my peer group, Bloomberg has taken a pretty huge reputational hit. I mean, sure, we all still read 'money stuff' because it's "the daily show" for financial news. It's great entertainment, and moderately edifying. But, you know, it's one of those things where after a newspaper writes about something in your field, you start wondering if they are that ridiculously uninformed about the things that aren't in your field.
So, I felt really pretty safe with buying in the dip.
Without commenting on whether or not I believe the Bloomberg story, I very much doubt that "people were tearing these boards apart", doing reverse hardware-level engineering looking for an unknown. Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule.
Doing the same thing with ICs and PCBs, without engineering diagrams and documentation, and needing very specialized skills and tools, must be 1000x more difficult. Maybe Supermicro and some intelligence agencies did internal studies, but they aren't going to tell you anything. Perhaps Apple and Amazon asked around in their engineering departments to see if anyone saw something. Possibly a couple security researchers took a look to see if they saw something obvious. But I don't think teams of experts were mobilized all around the world trying to pinpoint a deeply hidden hardware vulnerability.
I'd love to be proven wrong. Hacker News is read by ~50,000 engineers and programmers and related professionals a day. Admittedly my comment won't reach 50,000 people, but I'll ask anyway: Is there anyone reading here who personally analyzed a Supermicro board in any meaningful way, or knows anyone who personally did so?
You can parallelize this search, because the people doing it don't need to know much about what they're looking for—they're just hunting for visual differences. Unless/until one of them actually finds anything different. At that point, you need better resources to figure out what's different; but at that point, all hell also breaks loose, and you'll get all the resources and help you need to do better analyses.
Boards bought weeks or months apart might have lots of differences in traces and components due to engineering modifications and normal changes in suppliers. Do you really think that all hell would break loose if you showed that there were differences between a Supermicro board made in March 2018 and one made in September 2018? Other than maybe Supermicro or intelligence agencies (who aren't talking), I doubt that anyone has done what you've suggested. People are busy and it's a lot of work based on an allegation and no exact place to look.
There are examples of attacks which cannot be detected by staring at X-rays of components[1]. Now, to be fair they are very advanced attacks but we know they exist.
[1]: https://link.springer.com/article/10.1007/s13389-013-0068-0
I think... we might just live in different worlds? I'm a sysadmin and I maintain computers, but I am surrounded by serious programmers (I mean, kernel and system verilog types, not frontend people)
I have worked places where it was just me and the frontend folks, and in that case? yeah, It's kinda lonely when you need help on anything serious. But even then, with my social life being what it is now, I'd have people to ask if I was in that situation again at work.
I... usually don't have any trouble getting someone qualified to look at my backtraces when I have kernel panics, and I usually don't have trouble getting those people to give me patches. that requires analyzing the code.
I mean, sure, people don't spend a lot of time reading random code for no reason... but you give them a clue that something interesting is somewhere, and my experience? you can get plenty of help.
I can tell you people were literally tearing them apart; cutting open capacitors and the like (common place to hide extra components) pulling apart power supplies (as I recall, one of the bloomburg illustrations implied that something was in a power supply; I remember a long discussion of how you could compromise something over i2c from a chip actually in the psu) - I mean, sure, these things could be better hidden to the point where such techniques would be ineffective, but my reading of the bloomburg article was that they weren't particularly well-hidden, and that therefore they were discovered. Hiding that sort of thing from another EE who designs that sort of thing seems like it's probably pretty difficult.
(I mean, it is common to have compromised firmware; that's way easier. but that's also not what the bloomburg article was about)
But... yeah, I guess a lot of my assumption was based on my reading of the bloomburg article implying that this hardware was something tacked on in a detectible way; the sort of thing that could be done by some third-tier subcontractor. I acknowledge that if your attacker has the capability to produce ASICs that look just like the real thing and act just like the real thing out the same I/O pins, except in some rare case? yeah, that's not going to fall to this level of examination. but that wasn't my reading of the bloomburg story.
Furthermore, if an attacker went through that level of effort, they'd probably not just do it for supermicro. Most motherboards of the same generation are pretty similar; pretty heavily based on reference designs. If you were going to do something that sophisticated, that's where you'd attack.
If you were going to attack just one particular rev of one particular brand of a motherboard? well, something tacked on seems a lot more feasible.
(the story was that it was a Chinese subcontractor. Remember that SuperMicro is a Taiwanese company; I'm pretty sure that SuperMicro has less Chinese made content than any of the other major manufacturers of commodity server kit. )
I did know people cutting open power supplies, cutting open capacitors (a place chips have been hidden in the past) and otherwise examining the systems at the macro level.
I also knew people who managed networks of these things... and one of the bloomburg stories, as I recall, claimed that the hack was discovered via looking at the network as the BMC tried to phone home.
I mean, yes, you could do something more clever than that, replace a chip with one that looks identical, which uses the same traces that did something different, but the article certainly implied it was a more simple sort of thing.
I mean, if I was trying to compromise a BMC in that way, I'd just flash it with firmware that did what I needed (and this has happened before... to many different vendors) - my understanding of the bloomburg story was that there was some bit of hardware to re-flash the firmware to the compromised version after the user upgraded to a non-compromised version.
Likewise. Almost doubled my money before taxes.
My reasoning was that essentially no tech infrastructure company has significantly suffered financially due to security incompetence and it was unlikely that it would start with SMCI especially since it wasn't even being claimed that the company was in on it. Nothing even seemed to allege any particular error on SMCI's part, certainly not one that essentially every vendor isn't also making.
The idea that the report could be wrong wasn't a factor for me, only that the marketplace would continue to not care. (I find it hard to believe that it's entirely wrong: I am unwilling to believe that major state intelligence orgs are so incompetent that they are never backdooring hardware in that manner).
My guess as to why: one of their reporters was misled by a trusted source, or multiple trusted sources (who were maybe actually catspaws of some-group-or-other who would benefit from getting this story out.)
So now, although Bloomberg would like to speak truth by issuing a retraction, such a move would likely severely damage the original reporter's career; and they don't want to do that for something that wasn't that reporter's fault.
Having some corporation or government agency with tons of resources come along and force or trick a reporter into running a story, would be the journalism equivalent of an insurance agency's "act of God" waiver clause. I would guess that every major news outlet has a policy for protecting and defending reporters who—by hook or by crook—get used as patsies by forces out of the news outlet's control.
If you lie to a newspaper to get fake news published, you no longer have any right to remain off the record.
That's what a newspaper with integrity would do.
Cynical scenario: the trusted-source catspaws have been reliable sources in the past, and will seemingly continue to be reliable sources in the future. This is their “price” for giving up reliable information for free—the ability to be used as a mouthpiece by their organization. Bloomberg accepts this deal, because they’re so useful otherwise.
Noble scenario: the sources didn’t want to transmit this information any more than Bloomberg wanted to receive it, but they were forced to (by, perhaps, a National Security Letter; or just generic “we’ll hurt your family” coercion); Bloomberg knows this, and doesn’t want to hurt the sources either, when it isn’t their fault (and either don’t know who coerced them; or do know, but know that it’s an actor powerful enough that they know that reporting the source’s coercion would just mean the source’s untimely death. It’s not like a newspaper is a government that can offer witness protection.)
Spin is one thing, outright lies is another.
There are cases where the press really needs to take people to task but the amount of secrecy and ambiguity in this story makes it seem like a case where it can't (or shouldn't) be done.
>If you're not certain they lied to you, then your new story is potentially libel.
No, you report the truth that they said X to you, and your opinion that it isn't true, along with any verifiable facts supporting that opinion.
I have this view because FBI agents have acted upon information they could not fully understand or analyze before, only to be proven wrong later e.g. the arrest and subsequent exoneration of Dr. Xi Xiaoxing.
There were bunch of other things where I caught Tim Cook lying. Steve Jobs uses to be very deceptive when he needs to, his wording weren't clear cut and leads you to wrong assumption. Tim Cook doesn't have that talent.
And once you have been caught lying, you lose trust.
While I do have doubt on Bloomberg story, I am not going to trust sold on Tim Cook's word.
And if you have the means to do something like this, why not do it in a more subtle way that doesn't have people asking questions 6 months later?
I'm not suggesting you're wrong and I have no clue what happened.
What is Bloomburg going to lose here? Their reputation? They're already considered something of a gossip rag for financial news. FTC action? Unlikely.
uh, the pictures of a tiny rice-sized chip from the story were just photo illustrations. They were not supposed to be evidence of anything. It said so in the margins.
Or maybe you keep reading because you believe the other things, see the Gell-Mann amnesia effect, https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
I made similar money in the 2008 crash betting that certain banks would not fail.
So I asked the guy about them cause I thought it was funny that such a big company had such cheap looking marketing. And he said: “yeah our CEO couldn’t believe how much money some agency was going to charge us to design a marketing campaign, so he just did it himself on his computer in like 2 days”
I thought that was hilarious. But good on the CEO for realizing no one really cares what a server billboard looks like, and decided to save the $2MM or whatever someone was going to charge him for design. Super funny.
I love that old school Silicon Valley aesthetic.
Are those the $1M Silicon Valley homes I hear about?
This has already happened in a few places in the area. In Sunnyvale there are massive developments of million-dollar condos literally next door to mobile home parks. How much longer can those places survive?
One would think that the risk would make people less likely to buy. There aren't very many places in the area for people to move their homes to if the land they are sitting on gets sold.
The story was extremely interesting, but as people dug into it, it seemed like the reporters had a bunch of conversations like this:
Security Researcher: So, in theory, you can do a lot of crazy stuff! Embed a tiny chip on a motherboard, stick it in an Amazon datacenter, sniff all sorts of things...
Reporter: Are you winking right now? I swear you are winking! So you are saying this is true? OMG. Wow, what a bombshell!
Security Researcher: I was not winking.
Reporter: Suuuuuuuuuuuuure you weren't.
This is not new. There was something very similar in the late 80s and 90s, with everyone saying that Japan was going to take over, that they copy everything by sending teams in North America to take pictures of everything.
Authoritarian regimes can look appealing (from the outside, anyway) until circumstances change. Then they falls apart spectacularly because the commercial and governmental institutions are either absent, very weak, or are unable to adapt to the change. For example, the Soviet Union.
Problem with that idea is that it wouldn't just chase investors away temporarily, it would chase customers away who are a lot harder to get back.
> Server maker Super Micro is moving production out of China in a bid to allay US customer's concerns about spying, even though independent tests have shown no evidence of cyber espionage. The company has also announced its plans to expand its own in-house manufacturing facilities to help mitigate any perceived risks. A spokesperson for the company said Super Micro wants to be more self-reliant "without depending only on those outsourcing partners whose production previously has mostly been in China."
NSA Guy: Damn, we could infiltrate that target if we could get in implant in their boards but all their systems ship directly from china.
CIA Guy: Don't worry, I've got this.
I imagine it is nestled in some degree of truth, grant you, but I wonder if the goal is to force sentiment so heavily that vendors are essentially forced to go stateside. As in, did China suddenly want to exert state control over manufacturing done in China? Russia suddenly want data from Kaspersky?
[minor edit for readability]
The other thing we know is that every time Apple, Amazon, and Supermicro told the reporters there was no such chip on the motherboards, the reporters took this as evidence of a huge coverup.
Basically, the reporters believed what they wanted to believe, spinning conspiracy theories to explain the lack of actual evidence.
Because - as noted in the article - the practical effect of this story is that now people are worried about supply chains involving China, not that people are worried about Supermicro in particular. And selling this story on multiple fronts, official and unofficial, seems most effective.
And allowing negative side effects for domestic companies as collateral damage to harm other countries is standard practice; see e.g. visa policy. US companies can't hire the best workers, but it's okay because it helps the US government's political goals.
Whoa.
If China really has a chip that can reprogram/modify server motherboard instructions with only 3 pins all within the size of a rice grain (according to Bloomberg's story), they've already won the tech war.