Investigating Implausible Bloomberg Supermicro Stories | Hacker News Reader

Investigating Implausible Bloomberg Supermicro Stories

servethehome.com

112 pointsvanburen61 comments

> The point is, to combine even less powerful elements of each into a package with the functionality Bloomberg describes, it would require amazingly small lithography techniques. I have yet to find someone able to say that adding processing power, signaling interfaces, networking (even a MAC), and memory/ storage onto a package this size is even possible and I have asked folks in the valley that do this at major companies.

Meanwhile at the NSA, in 2008: "Not only can we do that, we'll throw in a high frequency radio as well for beating air gaps, and it will only cost you 10k! Oh, and we'll fit it INSIDE an ethernet port."

https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS...

I'm sure the 2018 series is even smaller and even cheaper...

That sort of thing is what stops me from dismissing Bloomberg's reporting. Intelligence agencies have a long history of creating implants that seem fantastical for their era. Consider The Thing and the Selectric Bug.

An implant small enough to go unnoticed on a motherboard somehow compromising the BMC to exfiltrate data or control doesn't strike me as the least bit far-fetched.

https://en.wikipedia.org/wiki/The_Thing_(listening_device)

http://www.cryptomuseum.com/covert/bugs/selectric/

OK, but Bloomberg said some Chinese spy chips were the size of tiny signal couplers. That "ethernet port" is actually a dual-stacked Ethernet/USB port measuring over 1 cubic inch in volume - it is hardly "small". 2008 NSA's "TRINITY" microcontroller, which forms the basis for most of their designs, is a bit smaller than a US penny, which isn't exactly "signal coupler" tiny.

I can't guess at what 2018 NSA might have, nor can I guess at what 2018 PLA might have, but I think it's hard to jump immediately to believing this.

With FEL lithography and enormous budgets most of the Bloomberg claims seem quite possible.

Why should someone attack a quit low priority target like Apple or Amazon with such sophisticated technology? It would make more sense to aim at say the White House or the Pentagon or something like that.

According to the Bloomberg article, Amazon learned of this when they did a security check on Elemental, a company they wanted to buy. Elemental's video processing devices with supermicro boards are present on US navy warships, for example.

And Amazon also runs the GovCloud. And are the leading vendor in the race for Pentagon's cloud contract.

Today everybody can buy a computer that is build into an ethernet port.

https://www.lantronix.com/products/xport-pro/

Yeah but this isn't news. Almost anyone who looked up the snowden leaks for ethernet bugs already knows this.

This isn't about replacing components. It's about dumping a chip somewhere and magically adding traces to the mainboard to connect it to critical components. Except this is impossible unless the motherboard was specifically designed for the purpose of being bugged.

This response appears to be grasping at straws to try and prove that the described chip doesn't and couldn't exist. It reads like FUD.

I find this article to be much more convincing https://www.theregister.co.uk/2018/10/04/supermicro_bloomber...

It points out that 6 pins is enough to intercept and inject code into the SPI bus between the Baseboard Management Controller and it's bootflash.

Once you have code running on the BMC, you can use it's capabilities to call home (via either the BMC's nic or the main nic, depending on how well the network firewalls are locked down) and then modify or simply read the main system's memory.

I don't want to claim that Bloomberg is right about the attack happening, just that the attack is surprisingly plausible.

Not to mention if you pulled this off, it’s likely you have many other hacks in place too.. such as defeating firewalls and the local network config.

They’re also taking bloomberg verbatim, when they likely have technical details incorrect. That doesn’t mean the gist is wrong (which may or may not be, I have no idea, I just think a lot of this article’s arguments are on weak foundation).

As a former spook with absolutely no knowledge about this incident, the whole thing sounds like:

* The Chinese supply chain interdiction / implant installation is real

* Those companies ARE aware of such events

* Those companies DID report it to USGOV

* The investigation is classified and Bloomberg didn't know or doesn't care

* The government is doing its traditional "Deny, disavow, do not acknowledge" etc.

* The companies were probably threatened with legal action / gag order preventing them from acknowledging the event

When I started working for DOD, one of the things in training was that you were always to say "I can neither confirm nor deny X". Eventually they realized that this statement itself is revealing, and suggested you just STFU.

The whole thing sounds incredibly plausible, and nooooooooobody was supposed to know about it.

The fact that Apple and Amazon are willing to go on the record adamantly denying it is more than just a gag order. That’s a serious breach of both SEC rules and public trust if it turns out they’re lying.

Does freedom of speech guarantee that they can't be coerced into punlic denial? Does a national security letter / other forms of government oversight override this?

(It absolutely shouldn't in a democracy, but there are plenty of things that shouldn't happen in democracies that have been happening lately)

Freedom of speech doesn't supersede securities fraud.

NSL probably does.

An NSL can compel you to not speak, it can not compel you to say something that is factually a lie in the manner they did. Look up the case law on “compelled speech” to back this up.

If the possibility of creating an international armed conflict could be on the table (i.e. infiltration of military infrastructure being seen as a premise for war), at what point would these institutions no longer "care" about what SEC rules are breached?

I mean, what would happen if this was seen as an act of war by the President's office? Surely there would be some above-and-beyond damage control measures.

Not to mention completely unnecessary.

If there is a conflict between national security and SEC rules, I am not convinced any court would go for the SEC rules.

Then again, I'm not convinced that these companies needed to be as specific as they were when denying bloomberg's story if they must comply with some court order.

That's not to mention the lack of evidence of an operation of this scale. Keeping that amount of people quiet for this long is something more newsworthy to me than spy chips.

Having been party to stuff like this before, absolutely this.

There are people at Amazon and Apple that know and are participating in an investigation. The vast majority of people (including almost all executives) don't know.

Amazon's denial for example came from the CISO of AWS (who is a VP), not Michael Cava the CSO of Amazon itself (who is an officer of the company) and the most likely person to be handling the situation.

Amazon's denial also came from Amazon Web Services CEO Andy Jassy, who wrote:

> @tim_cook is right. Bloomberg story is wrong about Amazon, too. They offered no proof, story kept changing, and showed no interest in our answers unless we could validate their theories. Reporters got played or took liberties. Bloomberg should retract.

https://twitter.com/ajassy/status/1054401346827243520

Given that the alleged incident concerned Amazon Web Services, I would hesitate to make the assumption that anyone knows more or would be more involved than AWS's CEO and CISO. Of course, you can accuse them of lying, but it seems unreasonable that they would both somehow not know.

I don't know much about Michael Cava but I'd guess from his LinkedIn job description that he's involved in Amazon.com security specifically and not AWS. He is not listed as an officer or director in the investor relations website: https://ir.aboutamazon.com/board-of-directors while Jassy is.

Cava's job as he described it on LinkedIn sounds like it involves retail operations security, and not AWS data center security:

> Serve as the Chief Security Officer for Worldwide Operations and Customer Service. Lead a team of security, loss prevention, inventory control and business continuity professionals providing physical security design standards, access control management, loss prevention and shrink reduction programs, inventory quality assurance, crisis management and investigative services in support of Amazon's global fulfillment, logistics, delivery and retail operations. (emphasis mine)

I assume that only the execs with (plausible) deniability can do the denial.

Much more sensible to attribute it to a stuff up by Bloomberg. Also, if they had this proof, much more likely this admin would use it as bluster against China. I'd be more inclined to believe the whole story is IO (i.e. lies) from the current administration.

This is an interesting piece, which may be correct on some points. But it gets off to a pretty rocky start, building its case around overly aggressive claims directed at uncharitable (some might say crabbed) readings of the Bloomberg piece.

> That first part starting with “telling the device…” is nonsensical.

A fairer statement, in light of the article's own explanation, would have been "assumes that the BMC is networked in an insecure way unusual (perhaps even unheard of) at large sophisticated tech companies, or a network compromised in such a way to bypass blocked egress routes."

In other words: no. The statement is perfectly sensical, and probably even true of many networks. The author simply doubts whether this could be possible in what he regards as a properly configured (and not otherwise compromised) network.

> The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works.

But the author then goes on to explain how BMCs could indeed be used to power on machines that were previously powered off, potentially allowing access to sensitive data. (Presuming that the machine is also compromised in other ways, presumably by downloading malicious code).

> Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely...This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed. We have discussed two patently false technical details in the Bloomberg article.

Er... couldn't a hypothetical backdoor BMC chip... turn the server back on? And then off again when it's done?

Since I couldn't flash my own OS in to the BMC components, where I work those ports and the 'combo' bmc+normal ports are not used on public facing networks.

A BMC really should be just a fully open minicomputer. Ideally there'd be a slot for something like a Raspberry PI to be used as an add-on BMC card.

I'd really like there to be more servers which support OpenBMC.

> That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false.

Based on my experience in the industry, I'd say this isn't quite accurate.

Yes, the author is correct in his assertion that BMCs are typically provisioned on private networks that are only accessible to outside users via a VPN. He is also correct that you cannot "tell" the BMC to do anything without access to that private network.

However, the author assumes that the operator has disabled egress connectivity on the private network setup for OOB BMC access. In reality, that happens less often than you'd think. Many firewalls by default do not block egress requests and without egress filtering, the BMC can still make outbound requests to the public world. A compromised BMC could easily "phone home" and receive instructions from a command and control server.

I've worked in a dozen or so Fortune 500 companies IT departments and I've never seen any incompetent enough to connect a BMC/management network to any type of Internet connected firewall, router, or egress gateway.

I've seen many that use flat management networks, which is a bad security practice, but connecting it to the Internet would require a new level of stupidity.

Sure, I've always segregated BMC dedicated ports to a non-connected network. But the BMC can also talk on the regular ethernet ports whenever it feels like it.

It can usually only vampire onto the first ethernet port. Depends on the motherboard, whether both ports are e.g. Intel, or one Intel, the other something else. Also, usually only 1000base-T not 10G.

I'm sure even 10baseT is more than enough to phone home.

The 1000base-T ports are often not connected to the production network, only the 10G.

If someone compromises even a single internet-connected machine and can use it to compromise its own BMC, the BMC network is now internet-connected.

Competent network security groups will configure the management network VLAN with filters that only allow traffic to/from the switchport & IP range of the management servers, i.e. no inter-machine traffic on the mgmt vlan.

Good shops will also disable all IPMI port traffic (623) on general vlans, as part of wider edge filtering (no egress of 1-1024, with exceptions for specific mgmt protocols like remote assistance, but only from specific origins).

In summary, BMC has long been considered a potential backdoor and been severely restricted for a decade. It is certainly one of the least interesting things you could choose to compromise if you were building a motherboard implant. More interesting things would include: RF antennas built into the PCB or ethernet PHY, but with plausible deniability, i.e. could be parasitic.

I've worked with mostly smaller to mid-size companies and they often connect the dedicated BMC ethernet port to a network that is firewalled but still has egress connectivity.

That's why I disagree with the author's assertion that this type of attack vector is completely implausible. It's entirely plausible if the network operator has not firewalled egress connectivity.

I think you might be confusing, say, an intranet at work, with how servers should be set up. My workstation should have pretty broad network access. Servers, largely, shouldn't, unless they have a specific reason for it, and in my experience they'll be blocked from the internet at large.

Nope, I'm talking about the dedicated ethernet port used for BMC access on a server. You would be surprised how often those BMC ethernet ports are connected to networks that are behind firewalls, but still have egress connectivity to the public Internet.

Dell servers with iDRAC(Dell's brand of BMC management) support online updates via https /ftp. Ours are firewalled but I've seen many that aren't.

That is truly ghastly. Dell (and others) keep adding this kind of rubbish and don't seem to pay a penalty for it -- presumably because medium/large corporate customers (not cloud scale) keep asking for it?

For HP iLO you have to pay for Advanced Premium Security Edition to get security features like firmware validation. You have to manage a fleet of licence entitlements otherwise your security stops working. Just managing that much licencing is a full time job.

iLO also has user scripts that can be downloaded to the controller -- what could go wrong ¯\_(ツ)_/¯. Lots of hardcoded credentials in the example scripts. It does at least have https, but managing the certificates is another big job.

Also, it's quite common that the BMC has a dedicated NIC, but also can use the main system NICs alternatively. A competent admin will turn that off (or not enable it, not sure what the defaults are), but why would code that has infected the BMC respect that setting? I don't see how it is enforced in a way the BMC can't circumvent.

People also assume just because the management network is isolated, that attackers can't get to it. The Snodwen leaks showed us that NSA uses radio transmitters to jump to hosts that relay packets to and from the network.

Someone with the sophistication to drop a chip onto a motherboard during manufacturing (which I don't doubt) and get it to the right customer could most likely slip a more advanced implant to bridge networks into a workstation headed to the NOC.

"Based on my experience in the industry, I'd say this isn't quite accurate."

I am genuinely curious, development or non development?

In this capacity, network engineering and operations -- mostly non-development.

But you don't need to be a network engineer to observe these problems out in the wild. Have you ever bought a dedicated server from a small ISP, asked for IPMI connectivity and they replied with a public IP address? Or the BMC was able to mount ISOs hosted externally on public shares?

>> Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off. (Source: Bloomberg with emphasis added to highlight key points for discussion)

This is not a very well-written article. How does this website's "best practices" document refute Bloomberg's story? The obvious problem is that not all organizations follow best practices, including many that you'd assume would, and those that do don't always follow them consistently. More subtly, if the BMC is subverted, you can't rely on to follow its normal programming or configuration: even if you have a segregated management network with no network access, the subverted BMC isn't required to use it and can use the "shared port" instead.

When you're dealing with subverted hardware or software, you have to throw out most of your assumptions about how those things work that were formed in normal cases. It's clear that the authors of this article did not do that.

exceptional claims require exceptional evidence, Bloomberg has not done so.

The article raises a fair number of solid criticisms regarding BCP to eliminate the described vulnerability.

This whole thing reeks. The only company that could be hurt by this is Supermicro and it's a small company, it reached 2B back in 2015 and has only declined since. This matters because if this is a con fueled by an SMCI short then such a short position would need to be so big it would draw the attention of SEC. Something this big, it's not worth for anything less than hundred(s) of millions and then you are sitting on a short position of like what, ten, twenty percent of the entire market cap of SMCI? This makes no sense.

The US feds are currently trying to persuade the Canadians not to buy Chinese telecom equipment for their 5G cell networks. A story like this gives some nice fear, uncertainty, and doubt.

The problem with this idea is that SMCI was delisted for not filing basic paperwork with the SEC, and the delisting occurred before the Bloomberg piece was released. [0]

[0] https://www.theregister.co.uk/2018/08/22/supermicro_facing_n...

https://www.marketwatch.com/press-release/supermicro-announc...

Right.

So then the SMCI short can be ruled out. What else could fuel this madness?

The story would not exist if not plausible. Even the many of the experts who dismiss the story say it is plausible.

I get that it is extrordinary, and requiring extrordinary proof as a result. However, extraordinary is not at all the same as implausible. There are examples of similar attacks in the wild, no matter how many orgs argue they are not affected.

I gave up when I got to the section on munufacturability. The idea that a package this size isn't big enough to house a large amount of digital logic is just wrong.

So it seems to me that it is possible the original Bloomberg article may not be true, or at least has gotten details considerably wrong.

In any event, I have a couple questions.

1. Is it possible that some kind of small chip could subvert there boot process of the BMC?

2. Can the BMC subvert the host system in some useful way that would be hard to detect? I'm thinking about altered microcode loaded at CPU powerup.

3. Can the altered microcode be used to subvert the host complex remotely? This could be something like a Spectre or Rowhammer attack.

1. Yes

2. Highly Unlikely, but can’t be entirely disproven from what I’ve seen.

3. See #2

"Bloomberg’s report describes an attack that is not possible at the companies listed in the article."

> Bloomberg’s report describes an attack that is not possible at the companies listed in the article.

Assumptions. This article is full of them.

Frankly, I'm not going to read this article or evaluate its claims.

But it's interesting. Apple (mainly) and Amazon and a few others seems to have successfully turned the narrative to the question of whether or not Bloomberg got played.

At that point it seems like Bloomberg has to put forward something a lot more concrete than "X anonymous sources told us so" or lose all credibility. (Or they can retract, though at this point, after multiple double-downs, it would would take a wholesale editorial clearout and significant time to return to credibility.)

If they could show us a compromised motherboard; and maybe even the Chinese tiny chip?

Reply on news.ycombinator.com