Making sense of the alleged Supermicro motherboard attack
lightbluetouchpaper.org
lightbluetouchpaper.org
FWIW at NetApp the firmware engineers called the BMC the 'BiteMeChip' because they always caused issues when bringing up a new filer motherboard. They were finicky, were often hard to update, and when misbehaving could completely screw up the system.
So I can see this as a vulnerability but really I'd want to stick some probes on that 'mystery' chip and make sure it wasn't just some LC filter or something which is shunting off noise on the data lines.
[1] https://github.com/ChuckM/stm32f469i/tree/master/demos/qspi
A chip holding and SPI line down - makes way more sense.
MISO is master-in, slave-out for anyone not familiar with serial peripheral interface jargon. Usually, the slave quad-SPI memory would send the configuration over this line, so pulling it low should dump the real data to ground.
[0] https://www.schneier.com/blog/archives/2013/01/the_eavesdroppi.html
[1] http://fish2.com/ipmi/river.pdfRealistically, in order to drive it low you don't have to bring it down to 0V. Most 5V chips will stop registering logic high around 2.5-3.3V for example.
(edit: 88 I/O is a modest size microcontroller sort of chip)
No chip design would ever do this. The chip would incinerate itself.
> superimpose a signal on the power-lines to notify the rest of the system
The rest of the system i.e. the BMC already know that something weird is up.
The SPI bus can be configured with shared master-in-slave-out(MISO)/master-out-slave-in(MOSI) lines since many SPI chips won't even drive the bus if their chip select line is not driven low. Thus the MISO and MOSI pins usually have a fair bit of buffering on them and will often be connected to the bus with a 1K resistor (either externally or built into the chip[1]). On a 5V system this limits the source current to 5mA. Either way these pins are designed to take some abuse.
Current drive is another issue because typically the output driver, if it isn't open drain, will use a p-type drive transistor which has a harder time passing current than an n-type transistor does. As a result the spec iOh (output current when the device is held high) is much lower than iOl (input current when the device pulls the pin low). So one pin to ground will overwhelm the output driver and pull the line effectively to a low state (not as low as it would if nobody was trying to pull it high, but low enough to read as 0). You will see this technique used in 'wired and' type circuits, where output pins are connected in common to a line, and any one going low will pull the bus low. If they are all logic 1 the bus is logic one, if one or more of them are logic 0 the bus reads logic zero.
[1] Yes I know that on "high speed" SPI ports this is not done because of the parasitic inductance in said resistor rounding out the edges of the data pulses with respect to the clock line thus reducing setup and hold time margins for accurate data transmission.
From 2016, https://arstechnica.com/information-technology/2016/03/repor....
> Apple has begun designing its own servers partly because of suspicions that hardware is being intercepted before it gets delivered to Apple, according to a report yesterday from The Information. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."
From 2013: “NSA reportedly intercepting laptops purchased online to install spy malware”
https://www.theverge.com/2013/12/29/5253226/nsa-cia-fbi-lapt...
For example, Elemental never used that board. See my tweets here for the actual specs: https://twitter.com/hugelgupf/status/1048160794565861377?s=1...
Elemental boards come with GPUs. The blade pictured has neither GPUs nor PCIe.
A year ago, Google announced their Titan firmware security chip[1], which would limit these kinds of attacks. I don't believe they designed and built this chip, and surrounding infrastructure, because of purely theoretical attacks.
Besides that, over the last couple years there has also been a lot of work trying to neuter the Intel ME, because of how dangerous it is. Another example is that Google had also done work to replace the EFI/Intel ME with a Linux kernel [2]. This has limited usefulness against the most recent attack, but it is related (since it's still a chip that has more privileges than the primary CPU).
I suspect that there are a very small number of people in these big companies who are aware of these attacks. It's hard for me to guess why the companies involved would deny that these exist.
1. https://www.zdnet.com/article/google-opens-up-on-titan-secur...
2. https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20wit...
http://articles.latimes.com/2012/oct/09/business/la-fi-court...
"Congress granted retroactive immunity to people or companies aiding U.S. intelligence agents."
In this case i'd not be surprised if Google/FB/AMZN/Apple/MS/Supermicro were even part of the CIA/NSA counter-sting - feeding of the false info back to Chinese intelligence through the detected 'mistery' chips.
> Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.
If they were under an NSL they would simply not comment on this at all. That would be pretty normal for Apple, so people would probably take it in stride.
You accuse me of selling pink and purple unicorns to gangsters. I reply that "I have no knowledge of any contracts or agreements relating to the sale of unicorns, horses or horse-related animals from my firm, regardless of the colour, breed or condition of the animals. I categorically also deny having any business dealings with any entity which has been charged on racketeering or any other gang related offenses.
The reality being that I 'rent' pink and purple unicorns to Don Corleone for absurd amounts, then don't revendicate when payments stop.
My rebuttal looks sweeping. I put in language to make it look like the scope of my denial is wide ranging and complete. In fact, I intentionally disclaim things you didn't claim to make it look like the moat of propriety surrounding me is vast.
But it isn't. I did what was claimed. And I didn't lie in my rebuttal.
Courts have surprisingly little tolerance for companies who think they're being more clever than their customers, their shareholders, or, for that matter, the judge.
You'll show they're acting in bad faith if documentary evidence shows they're baddies.
The theory behind our court system is one thing. The reality is another.
"No one from Apple ever reached out to the FBI about
anything like this," Apple writes. "We have never heard
from the FBI about an investigation of this kind."
Apple would have to move tens of thousands of highly specific CNC machines from China to somewhere else, set them up, and get their line moving again.
This is why I find the idea that Apple phones are "secure" to laughable on its face. China could kill years of Apple revenue if they ever did something truly offensive to the ruling party.
Apple would roll over post haste if China demanded it.
The ultimate source of most Chinese factory equipment is Europe and US anyway, under extreme political/consumer pressure electronics factories can be setup in a matter of months in US.
They would react as usual: lots of whingeing, no action, and roll over for a belly scratch afterward.
> under extreme political/consumer pressure electronics factories can be setup in a matter of months in US.
It takes 9 months to make a baby no matter how many women you get pregnant ...
Many of these equipment manufacturers have entire facilities dedicated to producing equipment solely for Apple. They simply cannot replace that amount of equipment in any timely fashion.
Edit (external reference): http://www.iphonehacks.com/2016/10/next-iphone-probably-wont....
> Koenig writes. “Apple is such a huge buyer of a particular kind of mill (BT30 spindle drill-tap centers) that Fanuc, Brother and DMG Mori each have factories dedicated to building machines exclusively for Apple.”
These sneaky chips were embedded in the PCB. So this is a supplier trust issue.
Plus they probably did not implant every board, So somebody inside FoxConn had to slip in these 'special' pcb's in known Apple mobo orders..
One side of me feels real bad for SuperMicro. The have always been there for the small guys that like to build our own servers and for small OEM shops.
Who else are the real server mobo competitors?
Gigabyte and AsrockRack from my view and they are a very distant second and third place.
If SuperMicro goes down we all lose and IBM, HPE and Dell gains.
“As recently as 2016, according to DigiTimes, a news site specializing in supply chain research, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan and one in Shanghai. When such suppliers are choked with big orders, they sometimes parcel out work to subcontractors. In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe. Eventually, that person says, they traced the malicious chips to four subcontracting factories that had been building Supermicro motherboards for at least two years.”
FWIW this is the principle behind warrant canaries. A warrant canary is the practice of putting a statement such as "we have not received any NSLs" in a regular report, and then omitting it once you have received an NSL. Because you've conditioned people to expect its presence, its absence then serves as a signal that you likely have received one (though not proof, as there are other reasons you may have removed it, such as being advised by your lawyer that it's a bad idea as the courts may not look favorably on the idea that, no, you weren't actually violating your gag order, you were technically saying nothing, given that you set up the conditions yourself such that saying nothing is in fact saying something).
If you have a sign that indicates that a secret event have not happened, the intent of removing the sign is to indicate that the secret event did happen.
The intent is particularly obvious to the originator of the secret event, so you won't be able to argue in the court that it was entirely coincidental.
Could you have a canary community that calls companies once/quarter and specifically asks the NSL question? Seems safer for the companies themselves.
This isn't true and they probably don't work in any other country in the world. They're explicitly illegal in Australia, apparently.
If I remove the sign just because I no longer want to keep the sign up and someone takes it to mean that the secret event did happen, when it really didn't happen, do they have ground to sue me for fraud or lying?
Neither grand juries nor coroners "outrank" the Executive (unlike Congress, who I would assume can just throw out the NSL to get testimony, since they have similar powers, like throwing out a document's top-secret classified status to get it read into the public record.) But in both situations, you can still be found in contempt of court if you just say "no comment."
It cites some hearsay speculation that Warrant Canaries are illegal, and cites several court cases that say Warrant Canaries are protected and compelled lies are illegal: "West Virginia State Board of Education v. Barnette and Wooley v. Maynard rule the Free Speech Clause prohibits compelling someone to speak against one's wishes; this can easily be extended to prevent someone from being compelled to lie."
Came here to say this. When I read Amazon’s rebuttal my gut said “what if these folks had to respond but weren’t allowed to tell the truth?”. If the allegations went unanswered it could be damaging to Amazon and tip the hand of the spooks. If they answered and said they did find the devices the story could run away from them, the internet mob is good at writing articles with “problematic” in the title.
Still I think this was a calculated leak. Why? Probably to put pressure on China in negotiations.
And the make or break for AWS is the assurance that when you're runnign stuff on one Amazon pizza box, your data are safe even from other users on that same box.
You HAVE to get your assurance from Amazon if you're a corporate customer, for due diligence reasons. If that assurance is established to be a lie, every single corporate customer they have would be obliged to migrate to MSFT/RHAT/GOOG or elsewhere. AWS would be dead.
I would say it's a calculated leak that happens to be materially false, meant to pressure CHina and kicj Bezos in the shin.
Read the denials carefully. They don't say the attacks haven't happened. They say they haven't found a variety of things.
The apple denial in particular is interesting as it indicates that they have been corresponding with Bloomberg about this issue for a YEAR. Yet despite the magnitude of contact they refer to, they do not describe the nature of the vulnerability being discussed or refer to specific material elements they've refuted. Their statement regarding Bloomberg's most recent version of the facts isn't on material elements of the story.
Having been involved in corporate risk responses I can say I've seen more vehement denials penned with perfect factual accuracy in respect of larger allegations, where the allegations turned out to be true. Almost every statement of defense looks exactly like this, with easy-win half-truth rebuttals to facts peppered around liberally to call the competence of your opposites into question.
Edit: inserted a missing word.
"Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement."
Also:
"If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement."
And:
"No one from Apple ever reached out to the FBI about anything like this, and we have never heard from the FBI about an investigation of this kind — much less tried to restrict it."
If you read those as not material, then I'm at a loss as to what exactly would convince you.
"Apple has never found [...]"
So what about third parties/reports/partners/contractors? Have they found anything and is Apple aware of those findings? Not disclosed here.
Are the QC processes in place sufficient to lead us to believe that Apple would/should have found this issue? etc. If not, who cares if they haven't found it.
Before you complain about the semantics, Apple specifically uses the phrasing "We are not aware of any" in respect of investigations, yet they don't use that all-encompassing language in reference to attacks.
"malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server"
What is a malicious chip - could the inserted piece be classified as non-malicious or simply not a chip? What is a hardware manipulation - are they saying they've literally never found any deviation between spec and what they've received? Why place 'purposely planted' as a qualifying requirement there?
"If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement."
They don't indicate what was claimed. They reference a volume of correspondence with a variety of claims instead of the claims. We assume, contextually, that they're referring to the specific chip insertion, but that's not borne out by their statement.
"No one from Apple ever reached out to the FBI about anything like this, and we have never heard from the FBI about an investigation of this kind — much less tried to restrict it."
This is fully plausible and means nothing. They restrict their denial of contact with the FBI to outgoing, indicate they haven't received notice of an investigation of some kind, which doesn't mean there's no investigation - collaborating with the FBI specifically on this point wouldn't involve the FBI disclosing the full scope of the investigation to them.
None of these statements actually deal with the issues claimed. They look like they do, but they don't.
Want a real statement? "Upon being notified of the potential issue by Bloomberg this past year, we have analyzed all of the SuperMicro boards referenced to determine if any unauthorized chip insertion occurred and have found none of the alleged inserted chips or any evidence of associated suspicious network activity."
You’re making up new assertions from whole cloth - the original story claimed Apple found the chips then alerted the FBI. Apple explicitly and clearly denies every single aspect of the BW/Bloomberg story, and your objection to their denial is that they didn’t deny actions never presented in the article?
That’s called a straw man, and it doesn’t pass muster...
If someone that is talking about bash called linux for "a unix", I don't think most people would support your stance that they made completely baseless statement.
Just for some perspective on how dismissive your comment is imagine the following statement being said in reference to your team: "What do you mean, you had a network breach? Aren't our ops people not capable of dealing with silly 1s and 0s??"
It almost feels like Apple tried to bait more information from Bloomberg to identify the person who spilled the beans.
And while it is indeed possible for the government to make you stay quiet, forcing you to lie is considered compelled speech, which is a fundamental aspect of current thinking on the 1st Amendment, and not something that is likely to change. No, not even when a three-letter agency wants it.
Also nobody at the either the NSA nor Apple cares about the sort of everyday world news you're mentioning. Maybe lay of the Tom Clancy for a while?
USA is one of them - China is best suited for inserting "extra hardware" on a board as in this example, but if you'd want to insert similar functionality as some extra stuff in an existing chip, then USA could arguably do it easier than China.
Also, South Korea and Taiwan have immense role in the supply chain and can do similar large scale things as China.
However, yes, the lack of supply chain influence does make similar attacks much more difficult for a bunch of actors which are otherwise active in tech security area - Russia, Israel, North Korea, UK, etc.
USA would be easier to modify servers shipping to US, target specifically at datacenters.
China needs to guess which batch of the contract manufacturing motherboard would go. It would be risky to leak the board tamper to everyone.
You can run your RISC-V cores on an FPGA if you’re really paranoid. Of course, you’d be sacrificing performance.
Xilinx was not.
Xilinx et al. are open enough about how their chips work imo. You are not going to find something like Intel ME on an FPGA.
https://www.marketwatch.com/story/super-micros-stock-set-to-...
And... it's just a hypotheses, assuming news of such an exploit is indeed being suppressed.
My guess is that they know that everyone's confidence in them would crumble. Every business, ever household, everyone, would suddenly be aware that their data is not safe, even in the hands of the ones who say "trust us, we'll keep it safe".
Why? Even if this were all true, the sum total of the accusation is that a small number of servers 4 years ago were compromised and Apple's security chops are so good they found a completely never-before-seen hardware attack and stopped it with the help of the FBI - all without any customer data being at risk.
> The modem represents a milestone for Intel in a couple of ways; it is the first chip to be manufactured solely in-house and it is Intel’s first chip to support CDMA and GSM.
> Apple’s original plan for the 2018 iPhones, via Nikkei, was for Intel to have exclusivity on modem orders for the first time — amidst its legal disputes with Qualcomm.
The only plausible scenario in which none of Bloomberg, Apple, and Amazon are knowingly lying is one in which only a select few employees at Apple/Amazon knew about this and were talking to the FBI, as you suggested. Except this doesn't make sense. The only way in which a select few employees would know about this and nobody else is if these employees are subject to a gag order. But the only way they could be subject to a gag order is if the government came to them in the first place, said "here's a gag order, now that you have it I'll tell you about the attack". But according to Bloomberg, a random spot check by Apple employees found the chip, and IIRC they said a third-party security audit ordered by Amazon found the chip on the Elemental servers. In both cases, this attack would have been immediately reported up to the highest levels at the company prior to even beginning talks with the FBI. There's no scenario in which an employee at Apple or Amazon was made aware of the attack, reached out to the FBI, and received a gag order, prior to notifying their superiors and having the information make it all the way up to the executive level.
Of course, you could claim that Bloomberg was wrong about how the chip was found and right about everything else, but that's not really plausible. Why would they be wrong about that and right about everything else? How would it even be possible for the government to have determined that Apple and Amazon had their hardware compromised without Apple and Amazon's knowing cooperation? The government doesn't have access to these servers, only Apple and Amazon do.
By having a spy in Chinese intelligence who sells it, then doing inspection when hardware goes through customs.
Now, I think all motherboard manufacturers - and especially high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. Under what circumstances is it possible that SM's QC missed this out on so many boards? Won't it affect things like the power budget, weight, and latency? What do professional EE people here think?
Also AOI might have picked up a difference.
I wonder if that's an actual photo of the chip in question.
> [...] a picture of the alleged implant. This shows a 6-pin silicon chip inside a roughly 1mm x 2mm ceramic package – as often used for capacitors and other so-called ‘passive’ components, which are typically overlooked.
The alleged component in question had six pins, if it were passive it must have been a bank or array of passives since individual passives generally only have two pins each.
But I'm not sure.
This seems like a very sophisticated attack.
But, I'm not sure but it seems like a bit of a red herring. Reading the article the chip in question doesn't sound passive at all: "The Supermicro board here appears to have a QSPI chip, but also a space for an SPI chip as a manufacturing-time option. The alleged implant is mounted in part of the space where the SPI chip would go."
edit: so maybe it was an array of passives that were meant to go across every pin of this protocol in a single package.
> ... In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says. Nested on the servers’ motherboards, the testers found a tiny microchip ...
Rogue insider, paid or patriotic, likely both.
> affect things like the power budget, weight, and latency
All three would have no measurable change, falling into measurement error margin. Its a death sentence for Supermicro, nothing will help to restore trust. They should publish a detailed post-mortem analysis, though.
More of an obituary at this point.
You can't put the toothpaste back in the tube
> ... high end server manufacturers like SM - use sophisticated automated tests and quality control on boards.
Not necessarily. Automated production tests are there to configure and exercise the system and confirm it works as specified. Such tests are good at things like finding bad solder joints, pick-and-place mishaps, misconfiguration of firmware and weeding out product that fails functional test. That, by itself, is hard enough.
Such tests are not _really_ able to detect things which no one is expecting, or worse, things which an adversary has specifically designed to avoid detection. Sure, if something gets "discovered" during a failure analysis, a test or process can be created to specifically address THAT problem in the future.
To find "unknown unknowns", you need an audit of some kind and that is definitely different from production test.
Typically this is done to check that BGA devices (https://en.wikipedia.org/wiki/Ball_grid_array) are soldered properly. Usually not done on every board, but only if there's a problem suspected. When it happens they tend to focus only on particular BGA components or suspect copper traces rather than the whole PCB.
However, if the source files already had the component(s) on it, it would pass all tests and QC.
The routine audit could have uncovered hackers in Elemental's corporate network doing things like ensure certain machines go to certain customers, etc.
Circuit wise, you don't really need to "intercept" (place in series with) the SPI lines. Two parallel drivers [0] will generally fight it out quietly, and you can guarantee yours will win by designing your implant to have the beefier driver (and perhaps only changing 1's to 0's, as P-channel FETs are generally weaker).
If there already was a footprint for an extra chip (debugging, part flexibility, etc), then it would not require modification of the board design. And an extra component can easily be hand soldered afterwards outside of the standard robotic component placing.
[0] The standard topology of SPI, but in proper operation only one is active due to CS lines.
Fair point about hand soldering, however it would be more obvious to the manufacturing employees for an entire production batch to be diverted to have parts added manually than to simply sneak in another reel to the hundred on the pick and place machine.
Thank you for writing your article! I'm currently a bit out of the security headspace and reading the Bloomberg article had me scratching my head like what are the actual details here.
Whose “law” is it that the closer you are to an event the more you can see that the reporting on the event is desperately flawed? This has almost always been my own personal experience.
To have the article hit the nail on the head with a entirely plausible attack means one of two things;
1) This is exactly what happened
2) A nation state wanted us to think this is exactly what happened.
Either way the NSA is involved. The people on the ground who knew it happened would have to be NSL’d from telling their superiors that it happened, so they leaked it.
Perhaps the denials from above are entirely sincere because the people who discovered it can’t pass it up the chain.
The only thing I’m having trouble with is that would have required either a low-level plant to have discovered it and someone else at a low-level found out and was sworn to secrecy, or a middle-manager plant intercepted the message as it went up the line.
The alternative, that this is a false flag, is pretty fascinating in its own right. TAO would have had to conceptualize the attack vector, and then someone planted the story.
Perhaps this is equally likely. In fact, this is actually an attack I assume TAO is using in the field, which would burn this attack vector presumably because they have an even better one.
Even more interesting, the idea TAO was actively using this attack vector, saw evidence in the field their attack was discovered (devices going dark) and so preemptively planted a story to blame the opposition before they accused the US of the same thing.
It could have been a real attack vector that didn't actually infiltrate the companies in question due to having been caught upstream.
But as "attempted attack" sounds far less sexy than "actual attack," the sources may have exaggerated the impact/discovery to further trade negotiation objectives with the spin.
In fact, the BS discovery part of the story might be an attempt to parallel construct a reason it was discovered to cover for a program that might do independent testing of hardware which, if revealed, would give attackers some sense of sampling methodology to be able to counter.
That would explain the vehement denials from the companies allegedly compromised and yet the realistic attack vector published along with the US entity bans on using Chinese hardware around that time.
idk, but this description of Gell-Mann amnesia seems to touch on that idea
maybe such a manager would, by doing that in this case, be protecting the top level execs
XBox was designed to function in the hands of the adversary, to be robust against peripheral attacks and even motherboard mods. Even the main memory was encrypted by the on-CPU controller. Obviously, no open JTAGs. A lot of expertise there.
In my fantasies it would form the basis of the DoD infrastructure and then trickle down to finance. It was deemed to be not solving any practical problems, so it didn't go anywhere. Oh well, I found other fun thins to do in my life.
The real problem is for stuff connected to the non-secure internet, like banks, industry, personal info, healthcare, etc.
Cool you had the idea, too, though. At this point, Id rather see someone just fund a Freescale implementation of the security parts on their communications processors. Can get more mileage in the market that way.
Further, going back to the original article, the majority of the information comes from alleged government sources, so not the people directly impacted, but rather those just helping deal with the fallout and coordination.
Assuming there is merit to the story, it will likely be some time before more details emerge, unless having the story out there now helps accelerate that process.
Supermicro assemble their products in US/Taiwan.
So in theory China need to accurately predict the pattern of how non-China factories install batches of the motherboard. I think it's extremely difficult to pull this off.
What's still missing in the overall story, is what Apple and Amazon denials mean. The Bloomberg article says Apple and Amazon independently discovered these chips (Amazon via a 3rd party) in 2015; but the blanket denials by both companies appear to deny this discovery and reporting to U.S. authorities. I'd argue it's unethical for a company to apply falsus in uno, falsus in omnibus to a PR statement, but it's plausible there is a sufficiently misleading or false claim in the Bloomberg article that they feel it's legitimate to dismiss the entire article. In the meantime, the company denials have to be treated as conjecture.
Where's the actual hardware? Why didn't someone decap the tiny chip and probe it? Its design should be well within today's reverse engineering labs' capabilities.
These things literally take time, and there's no indication of how widespread the targeting was.
In a sane setup, your BMC connection cannot access internet. You should build an isolated intranet for it (including VLAN or hardware isolation, not just subnet/IP), and put a VPN in the front gate. As a result, you login to your data center, or go to there if you like metaphors. If nobody’s there via VPN, BMC network is a silent and dark place. No connection to outside, no unknown traffic, just silence. Only exception may be the discovery packets of some BMCs, which can find similar servers and form federations for easier management. Even this needs some setup beforehand.
I manage lots of these servers for a long time, and this is my firsthand experience. :)
Attacks involving writing to / reading from memory & exfiltrating that data needs proof of concept code now. Hope someone with comparable hardware and BMC modifies OpenBMC's code to steal some stuff from the OS for research purposes.
If it can be proved, the results will be relatively fun and certainly revolutionary.
Having said that, it would be nice if networks were segmented in the way you describe. I've been appalled at the lack of segmentation I've seen in companies of all sizes that I've had gigs for.
TBH, while I'm knowledgeable about hardware, I'm a total beginner in attack side of cybersecurity.
At least, we are segmenting our networks like that.
Two pieces of the story that don’t add up for me.
I would assume the reason why no one noticed outbound traffic is because it’s dormant.
> Why haven’t the leakers just provided the actual chip? It should have been as easy as ordering one.
Sure, let me just order up a state intelligence compromise. It's right here on Alibaba right under Official_PLA_61398.
The attacker could use this to escape AWS/shared computing sandboxes/containers in order to attack their peers. Exfiltrating the data stolen could be easily hidden in something that looks like legitimate customer traffic.
> has to travel over wire as TCP/IP,
BTW, this is not the case. If exfil via conventional system networking is too hard to avoid detection, they'll find another channel. RF via LOS, ultrasonic, or some of a million other ideas.
Or if they can insert themselves into the maintenance team, they can retrieve their chips at a later date.
Or you can just use the default credentials Cisco is addicted to leaving in their code.
can you imagine how it would go down if a major corporation like apple experienced a hardware hacking based infrastructure breach, contracted a cybersecurity company, and the tech heading their case asked them like "well were you making sure to check which ports were open?"
The chip passes the stolen data between the VM instances by DMA. The stolen data hitches a ride inside an otherwise innocuous TCP packet storing log entries in that S3 bucket.
Logs are routine backed up off site.
There would be absolute nothing at the network level to distinguish the infiltrated packets.
Lower latency scenarios could be devised which would be similarly invisible but allow better command & control.
Original article supposes that the implanted chip was used to modify the boot process of the BMC in such a way that the BMC loaded its software over the network. The question is why this network activity wasn't easily detectable and firewalled off by default.
It seems that you suppose that the chip can be directly used for some sort of userspace-inside-a-VM to ring0(?) escalation. In particular, you suppose that the chip has access to main memory. Can you elaborate on what functionality do you think the implant could've had?
Outbound traffic is allowed unrestricted in 95% of the deployments, it is just a life fact, people trust their own systems.
My current deployments allow outbound SMTP only. All software packages (rpm or whatever) get pushed in via rsync from the outside, or are built in an adjacent lab behind the firewalls and pushed across.
Oh yeah - docker. I thought this one was pretty funny:
https://threatpost.com/malicious-docker-containers-earn-cryp...
Also, just because you don’t know where the boards are does not mean that they don’t exist.
This is telling you that your experience is limited, not that the story is wrong. Trying to do egress filtering at scale is extremely hard for all but the most basic threats. If they open a socket to data-collector.pla.cn, yes, probably a majority of large shops would notice that within a few months but what if it's just a connection to S3/EC2, buried in the noise of all of the legitimate use of those services?
Think about how hard it is if the attacker is smart enough to bundle that into other traffic: compromise your mail server and have it respond slightly differently to some spambots, have a webserver respond to the Baidu crawler with actual data encoded in the cookies it sets knowing that the Great Firewall can pick the data up (Baidu doesn't even have to be involved – just something which gets packets somewhere they can see them), etc. If it's on a network with people, that's especially easy – is that user-agent hitting sites in China a bot or just the staffer who keeps tabs on market news for that region? (But, you may say, our user and server networks are tightly segregated! Does that apply perfectly to your terminal servers? How would you know if your web proxy started making a few extra requests?)
The mention of seeing a problem first in a Siri data center was interesting to me because those data centers probably have very consistent network activity and it'd seem like the kind of place where the defense team would have the best chance.
There is no way that any large tech company security operation misses this traffic phoning home from a management vlan.
1. The implant can use the host network interface before the host OS starts 2. The implementation depends on VLAN tagging and the implant simply uses configures its network interface to use the same tag as the host interface 3. The implant can compromise the host OS when it loads and use its networking stack after it loads
If you deploy several thousand of those backdoor chips, you wouldn't set them up to ping your C&C infrastructure all on its own. Rather, you'd use a magic string as a trigger to activate the backdoor in some few targets to reduce the chance of the outbound traffic being detected.
IIRC the original story mentioned en passant that at least one company detected some odd behaviour on the network, which eventually resulted in hw examination and the uncovering of these moles. TBH, that part is immaterial: it might well be some parallel construction to avoid giving away NSA intel capabilities.
> Where did all the boards in question go?
Warehouse 51 [1] of course.
https://media1.fdncms.com/chicago/imager/best-approximation-...
1) Directly compromising via software the BMC is easy to detect if you bothered to look. (i.e. checksums don’t match), and would only last as long as the BMC isn’t flashed.
2) It’s simply currently easier to make a second component, rather than to integrate it into the main chip directly. Of course, if I was in charge of this project, I’d already be investing resources into exploring this option.
Why not just one with special code? It's not like anyone is routinely reverse engineering the BMC boot code.
It seems like an awful lot of provable trouble to go through (note that there is no physical evidence in the public eye yet) when you could do the same thing, at the factory, with just software.
Because AST2400 has input for two flashes, one main, one recovery. And the configuration that uses that is not common. That specific moment actually makes it harder to detect, while being very simple from technical side.
In this model, the implant basically contains a binary patch that is injected at boot time over a segment of the BMC binary - counter overflows, chip cuts out the SPI and transmits its payload instead. After the payload is injected the implant goes dormant again/resumes passing through the SPI, so dumping the ROM after boot, or even physically clipping onto the SPI chip will not reveal the wu-tang secret.
[1] https://www.apple.com/au/newsroom/2018/10/what-businessweek-...
Talk is cheap, show me the code/server/chip if they ever exist. Otherwise, the story is just a blunt lie fabricated by Bloomberg serving as a propaganda to bash China amid the Sino-America trade war.
Like seriously, why does my hobby consumer motherboard need that feature? Corp IT only ever deploys to large fleets of OEM machines.
HP's iLo could be got into if you used `curl -H "Connection: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA"`
Supermicro's horrid BMCs, of which there are many, all were horrendously lax in security, long before this chip was, or was not inserted. You didn't buy supermicro if you were worried about security, you bought them if you needed to stack stuff high, cheap and dense.
There is a reason why its best practice to put them on separate networks, with as much stuff between it and anyone else. BMCs are massive backdoors, and should be treated with caution.
Defenders simply do not think like attackers. If you're a defender it's lethal to try to think like an attacker. "I can't imagine how this would be useful therefore it must not be useful" are the famous last words of everybody who has cast doubt on a new and novel attack vector. As a defender you need to first estimate the unknowns and unknown unknowns, which over the last year alone have exploded (e.g. actual and potential Intel microcode vulnerabilities). If you needed this article to convince you of feasibility or even just practicality, please don't pretend to be capable of assessing the security posture of any complex system.
And let's be clear: this is not a new and novel attack vector. There are companies that have existed for quite some time researching and selling products to deal with this sort of attack vector. On the spectrum of hardware based attacks feasible today, this chip isn't at the complex end of the spectrum but rather the simple end of the spectrum. The complex end of the spectrum involves hiding logic deep within existing ICs, and there's ample literature to demonstrate feasibility of both implementation and detection.
The difficulty in pulling off these attacks lies not in the software or hardware, but the political, intelligence, and military apparatus of attacking countries. The economic costs of detection are huge precisely because, at the end of the day, fundamental security relies on trust, not technological hurdles per se.
First you need to run an air gapped network or at least switch enforced vnet for your bmcs. Three firmware on these is only updated every few years anyway, so they're likely full of security holes anyway.
Second, in general, outbound connections need to be monitored everywhere in your data center.
I am willing to bet that there are teams of spies who have infiltrated Google, Facebook, Amazon, etc. Spies from US, Russia, China, Britain, Israel, Germany, etc, must have dozens of spies working as engineers are getting access to all that data as we speak.
To think that they aren't would be rather naive, in my opinion. If I were head of the spy agencies in any one of those countries, it would definitely be the first thing I would do.
1) happen to be country that makes all electronics
2) Company with a lot of ties to your country is happen to be top server supplier
3) Companies that use these servers happen to work for DoD, CIA and all major points of interest.
But cpu, baseband, ram etc is certainly more serious consideration... And I don't see how you can get around that. Unless you plan to build a dumb phone around a Motorola 68k or something?
I.e., it's not.
The lesson of the Snowden leaks is clearly this: if it can be imagined, and it can be useful, and they have the budget, and it's remotely doable, then it's been done.
China almost certainly did this because they could. You only get one chance to do something like this, so you have to do it even if it risks losing the ability to do it in the future: if you never do it, then you never get the benefit, and if you do do it, you get to do it once, and once is better than never.
I think the real trick is a real covert channel that's hard to find both analytically and via stastics/machine learning.
I think the real trick is a real covert channel that's hard to find both analytically and via stastics/machine learning. Which is stil high enough bw.
For that matter, many a CS faculty at many a university have actually created their own OS. Nothing too sensational either.
> In 2009, a report presented to the US-China Economic and Security Review Commission stated that the purpose of Kylin is to make Chinese computers impenetrable to competing countries in the cyberwarfare arena. The Washington Post reported that:
> China has developed more secure operating software for its tens of millions of computers and is already installing it on government and military systems, hoping to make Beijing’s networks impenetrable to U.S. military and intelligence agencies.
They all have Linux distros, sponsored nationally.
https://en.wikipedia.org/wiki/Category:State-sponsored_Linux...
- Is a legitimate news organization incompetent when it comes to a deeply technical investigation?
- Why are large publicly traded tech companies refuting the article to the point of potentially defrauding investors.
I don't think the power of international players and the maturity of their strategies plays into it very much. I think you can assume they are incredibly powerful/pervasive and still doubt Bloomberg.
(I've never actually bothered interviewing with any of the above, maybe it's all entirely normal or you need to write your own self-balancing tree implementation regularly).
In order to intercept a QSPI bus, you would therefore need 4+1+1+2xpower lines = 8 pins.
This is not sufficient for QSPI.
Single SPI on the other hand requires Clock, Data, ChipSelect, Ground & Power = 5 lines, so that would be plausible.
(Yes, I know you can potentially get rid of the CS line, but that depends on what else is on the bus).
You harvest power when data is idle. CS is irrelevant. The rough clock speed is fixed, and you can match the precise timing from the data line. QSPI actually gets you access to data in both directions with the tradeoff of only getting one quarter of the bits.
Logically, you likely only need to recognize a few patterns that are each say 30 bits long. If you could shrink this down, it would look exactly like a boring pullup.
In fact I would think the most stealthy and robust way of bugging code would be to just ignore addresses and look for a context-free stream of instructions that does some security-sensitive initialization (either of the platform or of the application code). Turn off some flag that's supposed to be on, causing the code to continue running as normal but silently skipping certain checks.
Now that you mention other devices on the bus - those in fact would be a great chip to subvert with such a backdoor. Why bother fitting everything into a two terminal hack package when you could just have a malicious temperature sensor sourced from the "lowest bidder" that does the attack and jumps to its own ROM? (I think those are usually I2C but you get the point). Bonus points for only being triggered through some sidechannel - recruit a low level employee at the target, while they retain plausible deniability.
If your flash chip simply doesn't return the correct responses to switch to QSPI, most systems will happily continue communicating with SPI.
Can't we watch it in action? Probe it and run the system, watch for activity on the SPI lines.
Can we dissect it and look for how it might interact with the world? Does it have anything like an RF front end, or does it wait for some specific BMC/network activity before triggering?
But so far no one has.
Maybe this will open up more manufacturing jobs in the US.
(i’m asking not suggesting)
and I do realize the majority of people do have the BMC enabled, including me)
also the fact that supermicro still offers this jumper to disable the BMC might speak to the fact that they (sm) are not in nor had any knowledge of this issue (ie they were not “In on it”)
Doesn't ASLR[0] mitigate this?
[0] https://en.wikipedia.org/wiki/Address_space_layout_randomiza...
If activated, it will short the circuit destroying full system operations. I don't even think it contains any sophisticated digital logic inside.
Most businesses are in that gap – definitely not tiny, but definitely not able to just ignore remote management – especially if they have lots of physical locations and it's not practical to have 24x7 IT staffing at every branch office.
Management engine vulnerabilities have gotten a lot of hype over the last year or two but most of it has been marketing for security companies rather than a serious risk (e.g. what percentage of bugs required local network access, which should not be easy to get on your management VLAN). During a similar period, we've also had Spectre/Meltdown and the usual slew of software vulnerabilities so I have trouble with the conclusion that the answer is to stop using a useful tool rather than continuing the industry-wide effort to improve the level of security competency among development teams.