New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom
bloomberg.com
bloomberg.com
The "trojan ethernet connector" paragraph mentions similarity to an NSA implant, which appears to be this: https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS...
I'm now wondering if someone found an NSA implant and misreported it as Chinese. We're going to end up in the stupid situation where people are afraid to report foreign intelligence attacks because it's illegal to report an attack by US intelligence agencies, aren't we?
Could you expand more on that?
Beyond that Apple, Microsoft, Tesla and probably Amazon have sufficiently capable public (or past public) researchers working for them. But I doubt many would want the publicity one way or the other.
Huawei & ZTE have been alleged with these exact type of attacks, by the US government
They get FISA court orders that force american companies to attach their equipment.
https://www.washingtonpost.com/news/the-switch/wp/2013/11/04...
Additionally, the story quoted talks about how the UK obtained the data and gave it to the NSA.
Nowhere is the NSA installing covert implants. They just don't do that.
The CIA does that :)
Ah, so he pinky-promised? Well OK then!
You are taking the word of a spy? Did he say it wittingly?
Other NSA programs that Snowden revealed were covert hacks. https://en.wikipedia.org/wiki/Global_surveillance_disclosure...
PRISM was at first reported as some sort of direct access to the servers of certain American companies, but it turned out to be the code name for a joint program with the FBI for using FISA warrants to request data from those companies.
Specific example aside, it's worth talking about why this does happen. "Black bag jobs" can mean "we didn't get a warrant", but they can also mean "we got a warrant and still aren't telling".
Even given a court order, there's still a possibility that employing surveillance by fiat will cause somebody to leak, or modify how they handle data, or simply reveal information about what sort of surveillance tools a given agency employs. Given that a FISA order can be obtained without a defendant, getting a court order and then doing the thing secretly anyway gives a sort of "bowling with bumpers" advantage where the project is approved if it gets revealed, but also done without revealing anything if it isn't.
More disturbingly, there's also substantial evidence that the NSA attacks companies covertly in places where they couldn't get a court order. Taking a specific device out of the supply chain and adding surveillance before it's shipped to the destination is a warrant-worthy project. Setting up systematic physical vulnerabilities with a use case of "turn it on some time in the future to get something interesting" isn't in the purview of a FISA order, so if the NSA did do that it would have to be without an order.
You can see how this benefits the NSA; if the voice mail is outsourced to a foreign company, and the NSA buys intel from that company, it's technically not spying on US citizens, particularly if they're getting metadata.
PRISM
https://www.schneier.com/blog/archives/2018/08/backdoors_in_...
> Juniper has confirmed that an initial analysis of malware linked to the National Security Agency appears to affect its firewalls.
https://www.zdnet.com/article/juniper-confirms-leaked-nsa-ex...
Whether China or the US re-allocates your IP, you can expect a competing product made in China. That you might have a relationship with one of them probably doesn't change anything unless they actually think your firm is the best one for the job of maximizing the results on their tax base.
I mean maybe I'm wrong; would it make any sense that these Republics take private corporate property more seriously other parts of their Constitutions they have violated at least until caught?
The US makes the claim China does not and former president and CIA head George Bush floated corporate espionage as THE plan for handling the absurd costs of "intelligence" criminals after the cold war..
I'm always astounded that working in a competitive market seems to blind people to significant stated facts of the environment their market is operating in.
In nature, it might make sense to just outrun the weakest, after all, a bear has a limited appetite. But superpowers have unlimited apetites and will collapse like the USSR if they should ever expand slower than cancer.
That doesn't make sense based on the assumption that US telecom companies already cooperate extensively with US inteligence agencies.
He went to prison because he sold stock based on insider information. Regardless of the reasons for his trade, it was still insider information.
Interesting. This one seems to rely on the presence of the USB connectors for powering. So, basically, any ethernet port with only Ethernet connectors should be safe from this kind of attack? The only powering option there should be PoE but I have yet to see someone buy PoE switches for a datacenter...
It was a small datacenter but we had a bunch for PoE KVMs.
I know that some IPMI implementations can control/use the on board ethernet. There might be some hole there. Similar for the built in management on some Intel processors and nics.
I would definitely spot that device if it were on these boards because it was described in detail and there were some pictures of what it supposedly looked like.
A device like that is not on either side of the board and it isn't in between the outer board layers (where it would be much harder to spot, especially if the cavity would be covered by a ground plane on one side).
I am not saying it is impossible, it is just very hard to hide something like that once you know it is there. The only candidate spots left that I can not check without destruction is underneath some of the devices or inside some of the devices. That would be a different level of sophistication than the original article alluded to.
In case you missed it, there is an article posted today [0] that has this quote from "Hardware security expert Joe Fitzpatrick", one of the Bloomberg sources, regarding "the supposed spy chip":
> In September when he asked me like, “Okay, hey, we think it looks like a signal amplifier or a coupler. What’s a coupler? What does it look like?” […] I sent him a link to Mouser, a catalog where you can buy a 0.006 x 0.003 inch coupler. Turns out that’s the exact coupler in all the images in the story.
The original article has now dropped into the real of SF for me until they show a detailed shot of an actual board with a parasitic device on it. Until then this is a wild goose chase.
Thank you for pointing this out.
Since there are many flash chips fitting the same pin out, all it took was soldering a compromised flash chip (with firmware for the BMC chip) onto pads that are already part of the design to compromise the whole system without any obvious sign that the board was tampered with (because in some SKUs, both chips were populated).
I'm not trying to be adversarial, even if it's only a theory it's an interesting one, but given the amount of conflicting information we have regarding this whole mess I think it's important to be clear about what's pure speculation and what's been reported by people supposedly in the know.
If you wanted to "backdoor" motherboards that shipped with these BMCs, wouldn't it would be much easier to just install your own "customized" version of the firmware on them? It certainly seems that it'd be much more difficult to incorporate another device into the system.
[0]: https://www.aspeedtech.com/products.php?fPath=20&rId=376
[1]: https://www.aspeedtech.com/products.php?fPath=20&rId=440
Normally this wouldn't be worth talking about because most active chips are too complicated and too design/supplier specific to carry out an attack like this, but SPI flash is about as standard a footprint/protocol as you can get in EE short of transistors so if you ship a product that could be reprogrammed from unpopulated pads, you're opening yourself up to a large attack surface.
Honestly, after I read the latest BMC chip theory I was like: "Oh, shit. Have I done that?"
https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i0hZ31udMZ_...
https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i9VdsjZLS_P...
They really have no idea what they are talking about at this time and it's probably fluff.
Long story short, that photo does not show the device involved.
"Robertson was unable to produce photographic evidence of the chips in question, saying they were described to him by protected sources. Indeed, Robertson in September asked Fitzpatrick what a "signal amplifier or coupler" looks like, suggesting the publication narrowed the attack package down to that particular component. Fitzpatrick sent Robertson a link to a very small signal coupler sold by Mouser Electronics. "Turns out that's the exact coupler in all the images in the story," Fitzpatrick said.
https://appleinsider.com/articles/18/10/08/security-research...
I thought China was famous for extremely short turnarounds for industrial engineering edits, so it seems plausible that they could manufacture the boards in a reactionary way and not need to do much in the way of logistics to get them to their targets.
The BMC back then was by a company called ATEN, who make KVMs. The modern BMC is by ASPEED - I don't know if they're related.
What's described in the article is exactly how the old ATEN firmware worked normally. It was a spectacularly poor product from a security perspective.
It would also be great to know what the attribution is based on. Just the fact that they're manufactured in China? Who else might get their hands on these devices in the shipping chain? What kind of traffic did they monitor? I guess just observing it's talking to a Chinese address doesn't tell much. I mean, just take an S3 bucket and dump your stuff in there. Setting up your own server in your home country pretty much screams "we're here!"
> I'm now wondering if someone found an NSA implant and misreported it as Chinese. We're going to end up in the stupid situation where people are afraid to report foreign intelligence attacks because it's illegal to report an attack by US intelligence agencies, aren't we?
That's pretty tinfoily, but it'd be a cool way to still report on it. "Whoopsie, I totally thought it wasn't you guys, sorry for disclosing"
From the original Businessweek article:
"Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China."
I know that some refineries do direct delivery for some of their large customers, especially industrial lubricants and other by-products. If the order is big enough, or someone wants to pay the premium, then direct delivery could be very feasible for tech too.
I know it won't probably won't apply to general purpose motherboards or devices, but is there a way to design or build some components or devices in a way that you can verify that they can perform their purpose and nothing more?
If we start with that concept, and slowly build up "verifiably secure" components, they can be the islands of security that we can build off of without having to worry if the manufacturing plant left their door open one day and some random person was able to sneak in.
For a motivated and well funded attacker who has an ability to manufacture a replacement chip with an additional coprocessor that can siphon or modify data from the main processors, network cards, and baseband modems, short of decapping every chip and component that comes through your assembly line your resources would be better spent on establishing trust mechanisms with your suppliers and the transportation couriers touching your devices before the end user acquires it.
https://en.wikipedia.org/wiki/Tailored_Access_Operations#Kno...
http://www.spiegel.de/international/world/the-nsa-uses-power...
A way to verify a chip is working as expected in a way that it can't be gamed without breaking multiple fundamental proofs, so that you won't need to worry as much about who makes it, just that it "passes the tests". (and you'd probably need a system to validate the validators, but splitting up the people involved means it is significantly harder to hack multiple products to all have them falsely verify each other)
Obviously I have no idea what I'm talking about and am just kind of musing at the idea, but trying to secure the whole supply chain from digging materials out of the ground all the way until it is in the hands of the consumer seems like an exercise in futility. You'll never be able to secure it in all cases, and like you said a truly motivated attacker is going to be able to break the chain (even if it means threatening a handful of people with death so you can get 5 minutes alone with a board).
Already exists in the form of 'country of origin' procurement for high security applications.
Personally, as someone outside the US, I would gladly trust alleged Chinese malware over known NSA malware. Or even better, literally any other country outside the 5-eyes.
[1]: https://www.theguardian.com/books/2014/may/12/glenn-greenwal...
That's interesting. As someone who has bought hundreds of thousands of dollars of gear from Supermicro (and has been a huge fan of their products and designs) I always thought their chassis were their core product.
Recently SM started to go down the "you can't buy our JBOD chassis without buying them full of our qualified drives" ... I knew that was the end of the golden age (of SM).
Luckily this coincided with the introduction of the 60bay HGST JBOD chassis. We haven't looked back.
Yes, these units are stellar and anyone buying Supermicro JBOD units should be looking into these as much better replacements. If you have volume they can be even more competitive than Supermicro if you push.
<off to look up this beast>
This is a really hard story to know what to think about. On the one hand, yes, hardware implants are a major risk. And having so many of our electronics manufactured in a country with massive state control over its economy and with which we have an adversarial political relationship is definitely a big concern.
On the other hand, the denials from the companies cited in the first article are remarkably strong. And again this article fails to give relevant details. It just cites a security contractor who says he had a client who had this issue.
But Bloomberg is a serious news organization and they are holding strong on this story as well. So what to think?
It strikes me that if your goal was to ramp up tension between the US and China at multiple levels, then planting this sort of story would be a great way to accomplish it. Politicians can cite national security. Wary consumers are triggered over privacy. Corporations become more and more gunshy of investing in China and partnering with Chinese manufacturers.
I hate to dream up conspiracy theories. And yet, we live in a world where many states, politicians, organized crime groups, political groups, and corporations are all intentionally spreading disinformation of all sorts all the time designed precisely to ratchet up tension and suspicion.
I don't really believe that's what's going on just yet. But I also don't believe it's as straightforward as the Bloomberg stories make it out to be, either. Something very strange is going on.
Are they? The authors of this story published an unverified and in corroborated story about Heartbleed a few years ago, claiming that the NSA knew about it and was exploiting it (https://www.washingtonpost.com/blogs/erik-wemple/wp/2014/04/...).
No conspiracy theory needed, traditional media is dying, and in their last gasps of air they are destroying their credibility for the sake of clickbait articles without proper facts and coberation. They are being deceptive, because they know outrage and politically dividing stories are still working.
It's really said, but transparent. No way, Apple officially writes that rebuttle on their website if the story is true.
When the articles published by those "serious news organizations" concerns China, disinformation / lack of evidence are really common place if you carefully examine their source. I used to do so from time to time but grew tired of that
1. As a white-hat security researcher, you have an ethical responsibility to publicly disclose vulnerabilities after doing the necessary due diligence (informing the affected parties privately, and giving them the necessary time to respond, investigate, and come up with an acceptable solution).
2. As a US citizen, you can't report attacks carried out by US intelligence agencies.
I can definitely see the responsibility that patriotic duty would entail, but a citizen with no links to their country's intelligence agency being held responsible for the said agency's failure in maintaining operational discretion doesn't seem sensible to me.
Edit: update formatting.
Who says? Unless you've received a National Security Letter, a gag order from a court, or have a pre-existing relationship with the government that governs disclosure (e.g. security clearance), there's nothing preventing a researcher from disclosing lawfully obtained information. Stumbling upon a secret investigation doesn't make the information unlawfully obtained, even if you suspect it might be a secret investigation.
I suppose intent could be there if you share information about a device that says, "Warning: national defense injured if you disclose". But absent a duty I don't think a court would impute intent, especially considering the Free Speech issues (somewhat peculiar to the U.S.).
Notice that nobody has seriously suggested (AFAIK) that the journalists who assisted Snowden should be charged under the Espionage Act, even though their acts would seem to fit multiple provisions. I think that's because unlike Snowden they had no duty, which means the bar for the requisite intent and knowledge (i.e. whether something is really going to harm national defense) is incredibly high.
But who knows? It's a good point and it poses a ton of questions. Still, personally if I found a spying device on something I wouldn't hesitate to disclose it if it seemed noteworthy. I wouldn't feel chilled by the Espionage Act. The same law in some other country? Probably would think twice.
Sure you can. Short of a gag order (and maybe not even then) you can report intrusions all you like.
In any event, how does one determine the nationality of hardware that shouldn't be there? It's not like there's going to be a snarky "Designed by the NSA in Fort Meade" logo on the chips in question.
You also abide by a whole slew of laws regarding sensitive, secret, top secret, or SCIF information. If I knowingly, or even suspect, some information if classified, and I transmit it to anyone else than my federal assigned contact, I'm breaking major federal laws.
A lot of security professionals in the US have such clearances. So finding a NSA implant or such proof makes it dangerous to talk about by default.
So yeah, a gag order by profession.
> This claim seems like a big dilemma for US white-hat security researchers
It seems like the two are mutually exclusive.
https://dictionary.cambridge.org/grammar/british-grammar/pro...
https://www.quickanddirtytips.com/education/grammar/one-vers...
In terms of security concerns also - come on, we know by now to which lengths the US goes in this area, and they're surely doing worse stuff than this, I'd expect no one would doubt it any more. So, either they are genuinely surprised by this, which would be silly (a politically adversarial nation using an obvious opportunity - cheaper stuff being produced there for decades - and doing the same), or it's a part of a broader narrative that's being built.
And, to be clear, I don't think we (the world outside China) shouldn't be a bit worried given in what position _we've_ put China and how strong they are now - it's just that this kind of mass-manipulation and propaganda is the most-detested way of doing it for me...
They’re just more subtle, until they’re not. They have a very long term outlook on their efforts.
At least China won't be as dangerous as the US or the Soviet Union in that it has absolutely no interest in enforcing its political ideals on other places or becoming a world police. I see no real reason why people should be worried about the rise of China as if this will turn the whole world Orweillian. China has its own way of organizing the vast and complex country and it's hard to come up with better practical solutions. Still if one doesn't like it they can just live somewhere else, and the authorities don't care, as long as it doesn't hurt Chinese business.
Not yet. But becoming a banker for whole third-world is almost done. And when your debts are big, you lose sovereignty.
Cool. Let us test.
Error: Line 2:
> In my opinion, newspapers are all propaganda machines driven by some their benefit.
Most of these attacks never leave the room at corporate HQ where they are discovered unless an engineer wants to permanently screw themselves out of a career.
I once tried to leave a linkedin recommendation for a friend I'd worked with on a high profile project where he discovered Chinese state actors performing corporate espionage and we stopped it. The FBI came in and carted off the servers, we switched data centers, re-deployed, and that was that. We would never have been the wiser if he weren't closely monitoring network characteristics. 3 years and 2 job changes later he messaged me back to say, "Thank you for the rec. but don't mention that shit!"
Personally I'm confident Bloomberg's reporting is accurate to a high degree. Based on prior experience with investigative journalism there's no way they would go all in with a story like this if they weren't standing on firm ground. Every single sentence would've been vetted. For each statement made there would be someone whose job would be to reject it unless you could back it up properly. This is also why you don't see these entities defending their story against random criticism that pops up. Most if not all decisions have already been made by the time it goes public.
The fact that there's now a second story on the same topic is a good sign. The reporting of these things are usually followed up by additional pieces to increase the impact (and revenue of course).
They claim they have 17 independent sources. That's pretty impressive in itself. It also means that they probably worked real hard verifying their sources' claims and inputs. I find it unlikely that they would've acquired all those sources unless the thing was real.
And based on my prior experience I would make the exact opposite conclusion. Technical writers are rarely technical, and they seem to be happy to make stuff up and mislead - even if unintentionally - so long as they make their deadlines.
Every tech article written about a subject I was directly involved in has not even gotten the spirit of the topic remotely accurate, much less minute technical details such as what chip is used where.
That said - I fully believe supply chains are entirely compromised. I just don't think in this case I'd really put much stock into this specific reporting - they've already been caught blatantly misleading their readers by putting up a photo of a stock Mouser part and not denoting it as such.
I think a contributing factor is that it's generally hard to write about things you don't fully understand with the correct nomenclature. Especially when you might not be able to talk/ask for help about specifics with people more knowledgeable because of the secretive process.
Things could've been dumbed down, intentionally or unintentionally, by those involved. It wouldn't be hard to imagine a conversation like: "-So it was sort of a coupler thing? -You could say that, yes". Or what if the technical detail came from a Chinese source and Google translate mangled it?
The coupler thing is dumb and so is the picture (assuming it was a random product picture) but at least they might serve as a way of communicating the big picture: a hard to spot electronic "coupler" thing.
Maybe there is something there, and/or there is a reason to talk/substitute in vague terms, but insofar as the explicit technical details are concerned, they don't appear credible. Then you're left with an empty allegation that you will have to decide to believe or not based on no other ground than potentiality.
I do have to disagree about the competency of Bloomberg, though, they publish a lot of speculative, low-tech AI/ML scare articles that can be described at best as "inaccurate" and, more realistically, as "making stuff up". They used to have a good reputation, probably from their financial journalism, but their tech work is not good but any reasonable measure, in my personal opinion.
is it? and if you were witness to such an attack, how would you be able to attribute it to the USA vs some other actor?
Is this true? I mean outside a specific gag order or working under a clearance, you could find an issue with a piece of equipment, publicly talk about it and then be arrested because it turned out it was the US government who caused the issue?
Can you provide a citation for this statement please
How do a bunch of Supermicro servers vanish wintout anyone noticing? I’d expect quite a few people would be involved that do not have any clearances. Apple is known for their secrecy but a few other companies named are not.
Servers practically vanish every single day. Add a few more supermicro and it's not even noticeable. Business as usual.
2) You compartmentalize everyone so nobody has the complete picture.
Somehow, Amazon is really good at keeping secrets.
I say again to Bloomberg: picture (x-Ray) or it didn't happen.
Are you saying that Steve Schmidt, the AWS chief infosec officer didn't know about the hack? Or that his article [0] was published to purposely hide it?
If only one person in Amazon knew about it, it would be Schmidt. And if Schmidt knew, I don't think he'd write an article so strongly claiming Amazon doesn't know anything about it. The only thing in my mind that lends credence to Schmidt covering it up purposely is that $10B contract the Pentagon is putting out- perhaps they've told him to play ball as part of getting the contract. But even then it seems a stretch.
[0]https://aws.amazon.com/blogs/security/setting-the-record-str...
If you're attempting to hack me or steal data from me and I know you're trying (specifically as would be the case with this chip if the story holds up) then I'm in a much better position to try to figure out how, or provide misinformation, or try to turn someone in the chain of custody if anything needs to be physically handled. Or at the very least, if it's an espionage or military situation, it makes it easier to know who to kill.
All of that goes out the window if you immediately disclose every threat. Whoever is attacking you will simply use the means you haven't discovered yet and stop using the ones you have.
I believe you covered more in-depth content that could be filed under "buy the government time to investigate/respond appropriately."
Now you can observe them and only intervene when absolutely necessary, thus giving you time to learn more about the attackers and their methods.
If that story is true (and I personnaly think it has a high probability to be), what would a gov or a large org do? Investigate, confirm they have been compromised but then.... leave the hw in place and data flowing back to the alien mothership? Unlikely.
This issue in this thread is alleged to have taken place in August 2018.
In the intervening time, much could have happened.
Did the journalist and/or their friends and family make money on the massive drop in Supermicro stock?
Is the Trump administration asking to push this information out to earn favor in the trade war?
Are the investigators stumped and using this in an attempt to flush out new leads?
No idea.
This is exactly what I think. Anyone confirming such a case publicly could cause a huge international confrontation between two largest economies in the world. It's not about tech or business – it's about national security and international politics.
If Amazon is being spied on by foreign intelligence, wouldn't the NSA want Amazon to know about it? Particularly since government data is hosted on Amazon's servers.
The details that Bloomberg related previously are so different that this couldnt be what they originally were reporting on. This adds to the China hacking server board narrative, but it does nothing to prove the Bloomberg reporting actually true.
It does cast doubt on the denials made previously.
It seems this story isn't totally smoke and mirrors as Apple, Amazon and Super Micro seem to want us to believe.
Hard to imagine it remains an exclusive method by one actor.
It looks pretty clear that this is the hack in question, not an example.
Which is fun because you can accidentally put a super important insecure oob service on the same jack as an internet exposed web service.
Bloomberg only has second hand sources, and all the exploit details are based on speculation from security researchers -- not from insiders.
It looks like Bloomberg heard several rumors of supply chain manipulations, mixed that up with plausible scenarios thought up by security researchers, added a few photos from random electronic parts, and voila you have a compelling story to tell.
This "new evidence" talks about a completely different type of attack than the original article. It corroborates nothing. It just shows how misleading the original story was.
I think the most damning part was the use of so many misleading photos and illustrations. All photos were pure speculation (this is what this chip might look like, this is where it would make sense to put the chip). But neither the captions nor the text made that clear.
The only thing I believe about the story is that they have a couple of sources who have vague, second hand rumors about supply chain manipulation.
The bloomberg story stinks to high heavens even if there are live examples of infiltrated supply chains
1) Bloomberg has a number of sources that are mistaken/misinformed, but this is not necessarily a made-up story, or
2) Bloomberg is nearly correct (minus some technical details) but the US government is forcing these companies to respond as if the story is wrong - possibly because of diplomatic reasons.
What is the likelihood that #2 is correct?
(there are other alternatives, but I believe that the likelihood that this is 100% or at least majorly fabricated by Bloomberg is near enough to zero)
Would the US government have to force these companies to lie? It's quite possible that the denials were the result of voluntary cooperation.
A company has to follow court orders. The government would have to sue itself.
Now, a court order would indeed be something different, but I find it hard to conceive of a judge compelling a company to lie.
That's a nice government tender you're working on, looks very profitable! Would be a shame if someone rejected it because reasons.
While it isn't the direct influence described i imagine this scenario is highly effective and getting companies to stay in line.
The companies may have a strong incentive to cooperate in this campaign too, both to save face and government relations.
Unfortunately all of Bloomberg's allegations don't hold much water either, even though it'd be so easy for them to make the story credible with some details.
When it comes to companies like ATT/Verizon it seems like no forcing is necessary. They are complicit.
But they can. Maybe only in cases where silence is a canary, but that still sets a precedence for them being able to force you to lie.
I read a lot of (recent) American history books - #2 gets my vote.
So why is this suddenly breaking news? It bears resemblance to most of the propaganda stories we have seen in recent years:
- it is based on truth. Supply chain attacks are known to exist
- the US government has a goal of escalating with China over trade and IP practices.
- national security threats justify nearly any form of government action in today's world.
- the story turns out to have been leaked through foreign sources. This is typically the pattern we see because of concerns about propaganda coming directly from US government officials to US news outlets. There is typically a middle layer that is outside of the US where the allegations can originate from until they are broadly accepted as fact.
So Apple and Google are not really lying. Chances are there has not been any sort of major security breach in either of those companies due to supply chain attacks. It is possible that they have been barred from revealing information about it for national security reasons (in this case propaganda reasons).
So I think we can expect the following next steps:
- The story will continue to hover in this slow reveal format until enough laypeople come to understand the key concepts -- circuit assembly, components, trojan horse components, QC processes, subcontractors, etc. Once the stage has been set there will be more revelations and leaks from major companies that corroborate the story.
The goal is to make China the crisis in the buildup to the 2020 election. It's not a coincidence that this strategy is getting underway right after the midterm elections in the US.
Our president has already been attacking China with rhetoric and trade sanctions, and this story is meant to turn public opinion broadly against China.
The supply chain attacks do not have to have been significant (or successful) to make this happen. The very idea that "sneaky" Chinese intelligence agencies and firms would be able to slip this by US firms' quality control measures is enough to inflict paranoia on Americans and help them start to view China as a terrifying adversary that must be stopped.
China's military outnumbers the US military by 20:1 in terms of the number of active duty fighters, and China's economy is approaching first world standards in major cities far faster than the US had ever expected. China dominates scientific publications in the hard sciences, and its top universities are 10x more competitive (or more) than top US universities.
So hawks in the US realize that this may be the last opportunity for some sort of power projection or military driven containment of China's ambitions.
This is foolhardy, because China is led by a group of highly rational people whose policy responses to the administration's trade threats have been masterful and precise, and have conveyed with no uncertainty that China will not be bullied.
So what we're seeing is a short time horizon strategy by the US which is meant to have electoral consequences in 2020 and pave the way for some degree of hostile escalation with China. US weapons systems are still significantly more advanced, but China has likely weaponized many aspects of US infrastructure via these sorts of supply chain attacks. My guess is that the US Government is not aware of many of these, and will panic when they are discovered.
Fortunately for us all, China's leadership is calm and not prone to knee-jerk responses. China is rising to world prominence faster than expected, and the US will not take that lying down. However it is probably too late at this point, as China has a tremendous amount of soft power stemming from its importance to the US supply chain. Because of this there is still much hope for a peaceful, trade-driven equilibrium to emerge.
Ah yes, Bloomberg, well-known for protectionist rhetoric and support for Trump.
> top universities are 10x more competitive (or more) than top US universities. A citation would be very helpful here. And no, number of paper's published isn
> This is foolhardy, because China is led by a group of highly rational people whose policy responses to the administration's trade threats have been masterful and precise, and have conveyed with no uncertainty that China will not be bullied.
Citation also needed. They're so rational, they just detained the president of Interpol, have committed mass internment of Muslim citizens, and can't go six months without trying to ram NATO ships in international waters.
> but China has likely weaponized many aspects of US infrastructure via these sorts of supply chain attacks
Wait, what? So doesn't that mean the hawks were right to be suspicious? And that this really should be a national security concern? "This is all just a pretense to make people afraid of the PRC! By the way, you should be afraid of the PRC."
To be fair, many of Bloomberg's sources were government, so the government could easily have said "we'll leak this to Bloomberg." It doesn't require Bloomberg to have a protectionist agenda more than being a pawn.
I agree with you on the rest though; "masterful and precise" in particular seems pretty farfetched and ridiculous, and many parts of China's economy look like a house of cards.
China, Xi, and the Party are in a sense all one in the same thing. There is nothing more important than loyalty to the one Party, and the concept of one Party rule. And Uighurs are a threat to that, so they're labeled terrorists. If you accept the idea that one party rule is necessary, it's entirely rational to aggressively, perhaps even violently, ban all possible opposition. That's the nature of any autocracy.
“They do not want me or us to win because I am the first president ever to challenge China on trade,”
And converse to the calm of China, he is impulsive and has floated the idea of selective defaults on Treasury securities. And he has ample experience with this himself. It is in the realm of ridiculous conspiracy theories, but POTUS is a walking pile of ridiculous conspiracies theories, so why is defaulting on only Chinese owned securities more ridiculous than nuking Pyongyong? (Of course, only one of those can actually be contained.)
> China's leadership is calm and not prone to knee-jerk responses
Murdering people can be a completely non knee-jerk response to domestic issues. They ARE getting away with it, aren't they?
I guess I'm projecting some semblance of humanity onto them. I'd assume even the most evil would usually want to hold off murder till nothing else works. Hopefully karma is about to catch up.
Don't you think it's going to be suspicious when you see a capacitor with 6 pins? Don't you think anyone that inspects the motherboard is going to wonder why a capacitor has 4 additional lines going to a critical flash chip? Seriously.... The entire article here is around replacing entire components. Adding new traces to a finished PCB is impossible without using wire that will be suspicious to anyone doing a visual inspection. Most sabotage will probably just program the flash chips with malicious software without changing the hardware or swap big components that use standardised footprints like QFN with a nearly identical part. Adding a small capacitor sized micro controller where it doesn't belong and on top of that connect it to existing chips in comparison is extremely hard. At that point you might as well design the backdoor into the PCB itself and embed it between the individual layers of the PCB.
Consider what a state actor could do with access to modern microprocessor level fabrication.
I'd expect that we'd see features such as the following:
- sophisticated intra-chip communication
- long periods of total dormancy of the exploits
- circuitry capable of receiving a "it's safe to begin the attack" message
- surprising communications vectors for exfiltration
- technology to make malicious parts appear under x-ray to be normal
- fallback to awaiting the message to perform DoS if more sophisticated attack vectors are not possible
I agree with your suggestion about using the existing footprint, etc. There is likely some very sophisticated tech for making malicious parts x-ray and test as normal in every respect.
The network connector exploit described in the article would be easily detectable by temperature dissipation measurements. So distributed methodologies are likely in use.
I'd also estimate that a large number of mobile devices have built-in hardware compromises that are dormant and can be used if necessary. These would be the simplest attacks to carry out and would have extremely high yield. Things like:
- phones suddenly jamming the 4G and WiFi network simultaneously
- hardware implants to help detect whether a device is being used by a high value target. Such an attack could be created using a tiny bit of silicon and would be dormant in most cases.
The biggest risk to a state actor doing these kinds of attacks is being detected, so firmware based attacks are potentially more risky than hardware attacks, since we are better at detecting a checksum mismatch than we are at testing hardware across the spectrum of possible input conditions that might trigger unusual behavior.
So I think we'd see state actors dipping their toe in slowly to these kinds of attacks, first establishing the supply chain hacks without anything malicious going on, and then gradually phasing in actual malicious hardware once the relevant parameters for the attack are better understood.
I.e., They get credible reports of SuperMicro servers having been compromised, they know that Apple and Amazon are customers. They find 1 over eager source willing to say Apple and or Amazon received compromised servers.
What kind of conclusions the writers made from talking to the other 11 sources, what seniority they had, or even what companies they were from besides 3 from Apple, are anyone's guess.
I always am suspect when a government official or employee releases classified information. In some cases extreme moral outrage seems very plausible (Manning, Snowden.) The information is so startling the leaker decides they can handle spending much of the rest of their life in prison. In a much larger number of cases it is for political reasons (Libby, countless other stories where the source is never identified.) Many times, however, I suspect, the information is made up and then no laws have been broken, as far as I understand it.
Also, you can't really 'piggyback' ethernet easily, for the same reasons; you would need TWO phy in there to decode/reencode...
Even if you'd want to 'piggyback' on the link itself, it would be very, very difficult to say the least -- Gb ethernet is definitely not a gimme to synthesise, let alone piggyback.
So, color me dubious -- the SPI 'chip' of last week was a but dubious but doable (given not just a custom chip, but a custom PCB) but this ethernet story makes even less sense!
Still, if these are in the wild, then perhaps our chinese friends might have reduced the footprint even more to the size of one connector.
I know the connectors with integrated magnetics are quite a bit 'longer' and 'beefier' than the passive ones.
I'm thinking it's entirely plausible that such devices exist, and are broadly in the wild. Mostly targeted. That said, our own govt (US) is not innocent. Neither are China, Russia and many others. It's what government espionage actors do.
I think it's a fault of many that US mfg has fallen off as much as it has, and that critical infrastructure would allow foreign mfg in general. Or at least final inspection and assembly internally. Not just the US, but most countries.
Pretty much every BMC in existence (well, the ones that comply with the Data Center Manageability Interface, at least) can "piggyback" on top of an onboard Ethernet interface.
Regardless of if this is actually true, and so far that's debatable, clearly the rest of the world needs to retrieve ownership of their supply chains for mission critical computer systems.
It's hard to see how this doesn't embolden Trump to take more action against China. and to be fair at this point, its hard to disagree with him that China is taking advantage of the rest of the world.
Poor Super Micro. There time as a public company is probably measured in months now:(
> Appleboum said that he's consulted with intelligence agencies outside the U.S. that have told him they've been tracking the manipulation of Supermicro hardware, and the hardware of other companies, for some time.
I'd imagine at this point everyone who has the ability to do deep inspection of their hardware is working as fast as possible to resweep their servers.
One might imagine a cloud vendor is constantly the target from multiple state actors, foreign and domestic, all vying for universal access.
In addition--layering defense--one would imagine simply putting a motherboard on a quarantine LAN, simulating their production network, and watching its network traffic for phoning home.
The real implants might be waiting for a specific situation, like a date or a string on the bus, so you never really know if you got them all.
1. https://mobile.twitter.com/marcan42/status/10496904808065843...
And you have to do all that with a very low power device running from within the port itself. Seems like a very high bar to me, especially when there seems to be so many easier ways to backdoor a motherboard.
But maybe the component is only hosted in the ethernet port but is actually connected to other signals on the motherboard.
As to the beacon itself… DNS is pretty good. Just send an innocuous DNS request to a machine you control (say a NIST time server), if you think an iterative request won't show up on radar. Or send a recursive DNS request along a path you've wiretapped. (I'd be surprised if the NSA doesn't have a feed of all DNS requests to 8.8.8.8.)
Of course you will want to wait to see whether the bugged machine itself sends any such packets out first, to ensure that yours can hide in the noise. Bad idea to send a DNS beacon from a machine that doesn't ever make DNS requests.
Actually on second thought, given the above capabilities, you don't even need to inject packets at all. Just mangle existing DNS queries in such a way that you can identify them in a wiretap. Say, for all DNS requests with a specific hash, mangle the ID field so that it matches some orthogonal hash (and unmangle it on the way back of course). Very unlikely to be noticed by an IPS, and you can statistically determine that machines sending more than expected packets whose ID field matches this second hash are successfully bugged.
Or, why even send packets? Instead, drop all DNS request packets matching some specific hash. They'll eventually get retried with a new server or new ID. Again, statistics applied to wiretapped data can determine whom you've bugged. You don't even need store+forward capability here; just emit noise over the tail of the packet and the switch will drop it for you.
It doesn't mean it's false, it just means Bloomberg already cried wolf and couldn't show it. At this point it feels more like taking shots at Supermicro than professional reporting.
This story is incredibly weird, nothing makes sense to me. I can't believe that Bloomberg would willingly report fake news. The fact that they decided to go with it even though Apple and others told them several time using unambiguous language that they were not aware of such an attack means that they must have really trusted their sources since they're effectively saying "Apple is blatantly lying about such a critical issue".
But then why would Apple be lying like that? Seems like it can only backfire for them when the truth is eventually exposed.
Nothing makes sense.
https://twitter.com/RobertMLee/status/1049617855396933632?s=...
https://risky.biz/RB517_feature/
The podcast is an interview with Joe Fitzpatrick, one of the named sources. It's an impartial and mature discussion that left me with the same feeling of unease that Joe says he had when he first read the story. Basically, Joe would describe a theoretical hardware exploit to the Bloomberg journalist and the journo's sources would then confirm exactly that as a real world exploit.
Thinking about it this way helps me make sense of the topic. Maybe I'm weird.
I'm curious to understand if theres anything that can protect against this?
[0] https://cloud.google.com/blog/products/gcp/titan-in-depth-se...
> Based on his inspection of the device, Appleboum determined that the telecom company's server was modified at the factory where it was manufactured.
Any further evidence to support that claim? How can they be so sure that the boards weren't tampered with after they were manufactured and shipped to the states? Anyone with a little soldering experience could easily replace an ethernet port.
How uncommon are metal vs plastic ethernet connector sleeves?
This seems like a clue that even non-experts could use to track down an implant.
Reasons for an Ethernet jack to include a small metal enclosure:
1. The jack is subject to physical strain as we insert, remove, and tug on the cables. Most components are affixed to the board using only solder on the signal leads, but these do not provide significant mechanical strength. If you've ever had a bad headphone jack on personal electronics you are familiar with this phenomenon. So the little metal box has additional metal tabs that fit snugly in holes or slots on the board, and provide a stronger mechanical fit and a much larger surface area for solder adhesion.
2. Electromagnetic interference compliance. While Ethernet by definition involves pumping gigahertz signals out a long wire, these signals are carefully shaped and the cables pairs twisted to reduce leakage. But the designer of the jack doesn't know how much EM is flying around inside the case into which the port will be fitted. A metal box around the jack minimizes the size of the unshielded opening in the case. If you've installed a PC motherboard, you know the springy metal fingers on the backplate that seal against the block of external ports.
Is it possible the article is talking about an Ethernet cable plug? I have occasionally seen those with metal sides. But they are not normally supplied as part of a motherboard or server.
That would be a very, very good outcome, IMHO. Use espionage fears to push forward other objectives, like open source.
What makes you think that just because it's open source, that that would change?
i.e. the design is "Open Source" to SuperMicro - but it didn't help them.
https://www.digi.com/products/embedded-systems/system-on-mod...
or the 9210 version:
https://www.digi.com/products/embedded-systems/system-on-mod...
cannot exist in Gb speed, if I recall correctly the original Digi Connect came out in 2005 or so.
It also seems plausible the NSA would prefer techies not to look too closely at their hardware *removes tinfoil hat
All secrets eventually leak. All secrets. No exceptions. Not even nation-state players with unlimited budgets can prevent leaks. This is reality.
What's a hacker (in the original sense: programmer, not cybercreep) to do?
Our task as custodians of secrets for our end-users is to reduce the attack surfaces on our systems, slow down those leaks and mitigate the effects of leaks when they happen. We must do these things to the best of our ability. We must do them whether we rig systems for large organizations or for our grandmothers.
Who's trustworthy?
Apple? Probably. They're pushing security as a major component of their brand.
AWS? Possibly. They have a lot to lose if they're compromised.
Microsoft? Possibly. They too have a lot to lose.
Seventeen well-placed but unnamed sources in the US security apparatus, babbling to journalists? Possibly.
Journalists? Their trustworthiness is eroding.
Cryptographers Whit Diffie, Martin Hellman, Ralph Merkle, Bruce Scheier, Ron Rivest and colleagues? Likely.
Motherboard vendors? Probably not.
Router / switch / firewall vendors? Probably not.
Nation-states? No. (They could change this by abolishing "security by obscurity" in their work, but that would require major changes in mindset.)
So the US doesn't exactly have clean hands. Here's a theory:
1. The US wants to have more ability to spy
2. With most manufacturing in China, this is hard
3. Implicating Supermicro might cause their customers to switch suppliers. It has already caused Supermicro stock to go down 41% then 27% again.
4. The US / Sepio Systems / former Mossad / former CIA officials come to the rescue with a company manufacturing "secure servers" not made in China. Or more likely, still made in China, only under different control.
5. Result: profit, increase US ability to spy, "bring manufacturing back to US companies"
IMO, governments - all governments - have no ethics and will do whatever they want to further their agenda.
What is up with Bloomberg?
> The security expert, Yossi Appleboum, provided documents, analysis and other evidence
> ... said Appleboum, who accompanied them for a visual inspection of the machine.
The Trump administration has been consistently escalating retoric against the Chinese, and it's not hard to imagine the CIA/NSA/etc intentionally leaking facts to bloomberg that would make China look like a national security threat. This could even be done in a way where the security agencies don't leak anything actually false, but let the non-phds at bloomberg run wild with speculation to create a sensationalist story that's not really true. A recent WSJ article [1] has called particular attention to the Trump administration's escalating anti-Chinese rhetoric, calling it the start of a "second cold war". We know for a fact that these sorts of operations happened during the first cold war [2], so it's not at all hard to imagine they would happen now.
A false flag attack fits with all the information we have so far about the event: There's no direct evidence of the attack, and if the bloomberg article is ever proven to be false, then only a small number of security researchers (and HNers) will ever learn about the retraction. The vast majority of Americans will only remember reading about how "China hacked major US companies" and create an anti-Chinese atmosphere that will help fuel future anti-Chinese policies.
[1] https://www.wsj.com/articles/mike-pence-announces-cold-war-i...
From my lay understanding, Supermicro has a slam dunk case given the impact on their stock price.
At least in the US, I don't think this is true. Defamation is a broader class that includes libel, and always requires proving that the publisher was at least "negligent" in publishing the false statement: https://www.law.cornell.edu/wex/defamation
is it connected to the SPI of the BMC flash/OS storage? Why would software integrity checks like making sure the image is signed and not tampered wouldn't capture it? (Answer to this one sounds easy bad security practices regarding firmware process)
Curious how this one affected the server or did compare to the original article.
This one seems more benign considering it won't be able to mess around like the BMC has access to things like secure boot and other system busses like the PCI.
Does anyone doubt that semiconductor/electronics manufacturing will become a strategic industry in the same realm as uranium refining?
I think we'll see electronics supply chain integrity play a growing role in major project delivery in the years to come.
So I'm having doubts that Supermicro's Chinese CM has any idea who the end customer of any of their boards is. Maybe it's different for really huge customer, but I still have doubts- it implies they are building to order and have no inventory.
This implies that the CM installs the Chinese Spy Chip on every board, or a random sampling of boards (so they should show up in the wild) or the tampering happens later in the supply chain- in the US.
Also I have further doubts about embedding a chip in a middle layer of a PCB. I doubt the CM is going to add that much extra expense when it's easier to add a chip to the surface. You don't really buy much secrecy from embedding a chip given that it's easy to x-ray a board, so why bother?
Anyway, there are devices called optical comparators. Supermicro could buy some security by providing comparison images that allow customers to perform an incoming inspection. I'm thinking they should do this now to add assurance / help their stock price.
Here is an optical one:
https://www.visionxinc.com/digital-optical-comparators/pcb
Here is an X-Ray one:
http://www.glenbrooktech.com/multi-layer-pcb.php
Maybe the X-Ray inspection could pick up the Ethernet connector attack.
i dont know sometimes i like reading into the big picture, what is the purpose of this bloomberg article for those outside HN? it won't invoke feelings of calm but rather moral panic. now half the americans are riled up to think china is attacking usa, and we remember they need just teh right amount of support.
also as the noose is tightening around trump, a war or a limited military conflict would be the perfect distraction.
/end conspiracy
> Three security experts who have analyzed foreign hardware implants for the U.S. Department of Defense confirmed that the way Sepio's software detected the implant is sound. One of the few ways to identify suspicious hardware is by looking at the lowest levels of network traffic. Those include not only normal network transmissions, but also analog signals -- such as power consumption -- that can indicate the presence of a covert piece of hardware.
Ideas what they mean? I'd be surprised if mainboards etc exposed power consumption of random periphery parts. Maybe sleep/link states of the actual NIC that get influenced?
> exposed power consumption of random periphery parts
Sampling these analog signals is done outside of the computer itself. Even if it were exposed via i2c, you couldn't trust anything that it would tell you.
I've never actually checked in the bios of any server, as that information was never relevant to me, but they probably have that as well
Even if you find nothing you would then have a pile of strong evidence for the null case, eh?
And don't call me kidding. ;-P
Is this Bloomberg's new slogan?