Bloomberg’s ‘The Big Hack’
daringfireball.net
daringfireball.net
In response to the report, an Apple spokesperson denied there was a security incident, stating: "We’re not aware of any data being transmitted to an unauthorized party nor was any infected firmware found on the servers purchased from this vendor."
However, based on sources from within Apple, Ars Technica claimed Apple employees did find compromised firmware in Apple's design lab. Super Micro SVP of Technology also reported Apple terminated its relationship with them.
I believe we are seeing the same situation here.
[1] https://arstechnica.com/information-technology/2017/02/apple...
Read it closely: The old statement said that no infected firmware was found on servers purchased from supermicro, meaning freshly purchased servers delivered from supermicro were clean. The ArsTechnica clarifies in an update to the article that the infected driver was installed later.
The new statement says: "That one-time event was determined to be accidental and not a targeted attack against Apple."
The key technical detail of what these chips are allegedly doing also does not make sense. From the article:
the chips allowed the attackers to create a stealth doorway into any network that included the altered machines
How can you get around a firewall by using a compromised machine that's part of the internal network?
I don't think Bloomberg reporters are just making stuff up. But the technical confusion here makes me suspect that the government officials who leaked this story just didn't understand the details of a real incident that happened, and in the leaking the story got mangled into inaccuracy.
Fitting any chip capable of exfiltrating a nontrivial amount of data onto a modern motherboard without going through many rounds of simulation or significantly impacting performance, while also putting it in a place it is capable of intercepting valuable data is practically impossible. Hell, just getting the right power domains wired to the chip is going to be tough enough.
Edit: Based on other posts in this topic, it appears that the chip wasn't embedded in the PCB - it was just placed on top of an empty footprint for recovery flash. If that's the case, then this hack is technologically uninteresting.
I hadn't read the detail about the second flash chip. Obviously, if they're just putting compromised flash into a footprint left empty by the manufacturer on purpose for recovery, it's much easier, cleaner, and can be done with a small group.
>The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
Is your OOB-management VLAN/network firewalled off from outgoing access to the internet? If not, this chip can make requests out to another host, who gives it code to run. This code can then either collect data and send it back out, or do other nefarious things.
Edit: Now that I think of it, a compromised BMC can inject payloads into the running OS, depending on the sophistication of the attack and the defenses of the OS. So it does seem possible.
If true, it's pretty bad. For one thing customers will be more wary of buying hardware from China. If they move manufacturing elsewhere (or, for Amazon, source hardware elsewhere) they would have higher costs and lower margins, if they don't and a competitor does they lose business.
On top of that, people are already wary of the privacy implications of these companies having all your data, but most are willing to trust major US companies. If it looks like they can't keep it secure against random Chinese companies engaged in industrial espionage, more customers will conclude that operating servers in-house is the better choice.
It wouldn't be that surprising to see Supermicro move their production back out of China after this. Though they've been having a hard time of it recently already -- something about getting delisted from the NASDAQ. I wonder if the two incidents are related at all?
I don't buy it as written. There's something else here.
I agree it makes little sense to attack and deny rather than just stay quiet or keep it to a minimal PR blurb. But I think you're overestimating the amount of blowback they would get if they're lying. Most people just wouldn't care. It might be the decisive factor in a bigger company deciding where to put something new and them going to Oracle Metal or something, but I am really skeptical their bottom line would suffer because of this one thing.
Maybe AWS and Apple are counting on the top-secret nature of all this to enable their lies. Would US intelligence agencies really come out on the record to contradict the denials and confirm the hack if it meant destroying the credibility of a couple of American companies? The Bloomberg article detailed how the government was not willing to sound the same alarms against an American company as they were against Huawei and ZTE for just that reason. I'm guessing that creates enough ambiguity for false denials to be effective.
I’m not saying it’s guranteed that this story is good—-for one, I find the glossed over technical details a little questionable. But it is definitely plausible that this is a false denial, under orders or not.
Regardless, two points are very clear. Remember Stallman who uses that old X60 with a free BIOS? We all called him crazy when he insisted on not having management firmwares and secretive back doors. Now we have a story claiming they were used in an attack by a US adversary (as well as a long chain of confirmed recent security flaws). The second point is that we have to stop trusting China as the sole source for all electronics. It’s a national security crisis that there isn’t a single US facility making many of the components needed for basic computing platform. The federal government needs to immediately and strongly incentive more domestic foundries. And not the trump style “here be factory, jobs good” but actual targeted planning to bring the whole supply chain back, not just the final assembly everyone talks about.
There could also be simply a picture of the spy chip! It's pretty easy to have stronger evidence than this article has.
I think many comments have gotten far too conspiracy theoryish, but I can't think of any plausible answers to "If they were lying, how would we know?"
Reverse shell depending on how you have the firewall set up. Most firewalls don't block outbound packets, and even if they do you can tunnel over an open port.
> If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg.
If you are a major company, are you going to use AWS if you believe their hardware is pwned?
It makes a perfect sense to protect from a potential devastating blow to stock price.
China (or the NSA) hacking into something is non-news. Everybody expects that, especially since Snowden. It's their job, and the news would be them failing to do it.
And yes, I had the problem with my Lenovo desktop.
It could probably make more sense as a backdoor when the hackers get hold the access to the physical device, it is however quite incredible to me that this hack is designed for a remote network access, and could go unnoticed from infosec within the company if it is truly sending packets outside the firewall...
Nation states have the manpower for spray-and-pray attacks. They have less need for precise targeting.
> It could probably make more sense as a backdoor when the hackers get hold the access to the physical device, it is however quite incredible to me that this hack is designed for a remote network access, and could go unnoticed from infosec within the company if it is truly sending packets outside the firewall...
How is it incredible? Infosec isn't perfect. Equifax was hacked in part because their IDS system was offline due to an out-of-date certificate [1]. That system would have caught the exfiltration of data if it had been active.
Also, Amazon AWS must have all kinds of crazy traffic flowing through its network due to its customers. My gut feel is that it would be incredibly hard to characterize that traffic in order to proactively detect many kinds of nefarious traffic. The Bloomberg article stated they were easily able to find the traffic from these implants...but only after they knew what to look for.
[1] https://www.bankinfosecurity.com/postmortem-behind-equifax-b...
The Washpo digging on Roy Moore, the NYT digging on Clinton Foundation, etc. these are experienced reporters having 15-30 credible sources, evidence etc. every time this happens, everyone denies and slowly little by little the story finds more evidence and facts and it becomes true.
This is why a free press is soooo important
Why would Apple disclose that millions of their products could be hacked and you’ve lost all your privacy. Who would trust them? They’d lose billions, regulations would come etc. it’s in the best interest for every party to deny.
My suspicion is that there are many cases where the Chinese government is actually trying to insert backdoors into things, and that in particular Supermicro really has been compromised by the Chinese government, but the technical details of this chip are incorrect. That explains why so many government officials are eager to leak information to Bloomberg, but at the same time the technical details don't really make sense.
It isn't really in Amazon's and Apple's best interest to lie about this. When Gmail got hacked by the Chinese government, Google was pretty honest about it. China has a lot of resources so you can't really expect companies to fend off 100% of attacks on their own; it makes sense for them to acknowledge this publicly and get help from the US government when needed.
The current Newsweek is no more formal journalism than Gawker is, which is to say occasionally they'll get big scoops but most of the time they're in the mud trying to get clicks/eyeballs.
That hack involved several companies, most of whom were not up front about it.
I'm not saying this particular article is incorrect, just that I wouldn't strongly assume either direction without more information.
Edit: I absolutely could be wrong and things could be disproven in this, but just in my experience highly detailed articles like this are usually right on the money or close but just like everything in the world, can’t say anything with certainty.
> Why would Apple disclose that millions of their products could be hacked and you’ve lost all your privacy. Who would trust them?
You've got it perfectly inverted. This is exactly why Apple would come clean if they've been hacked by a nation state and take further steps to protect their customers.
Case in point: This happened to Google.[2] Can you imagine how much trust Google would destroy if they vehemently pretended that the hacking never happened? As opposed to going public ASAP with an action plan, which is what they did -- and it improved trust.
These conspiracy theories are never very well thought through. Conspiracies leak. Remember how Watchmen ended?
[1] https://theintercept.com/2015/04/10/twelve-years-later-u-s-m...
The American press is not perfect, but over the years I've been really impressed with the journalistic skill of our major publishers. Just this week the New York Times had an extensive story regarding the president's tax returns, they claim to have access to hundreds of thousands of documents and spoke to dozens of sources. I'm more inclined to believe Bloomberg's journalistic integrity than the denials of Apple, Amazon and China.
This is sort of like claiming that you have root access to Google's servers just because you have a Gmail account.
So the companies could deny that, but there is still a question of exactly what was (is) possible using whatever API they exposed, and what sort of legal review the companies do of the requests they get, and whether this review restricts what the government can do in a meaningful way.
This is exactly it. There's no cost to deny, deny, deny until it's impossible, then "oops, after extensive investigations we've found the claims to be true on a limited number of devices...".
There's a cost to damaging your credibility.
Don’t put too much emphasis on big Corp and their “credibility”.
As mentioned above, 1) these denials can be later parsed and opened up as necessary with significantly less reputational damage, and 2) they contain some interesting specifics that can make them strictly true but quite misleading, e.g., "Apple has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server" might more accurately read "someone else found these..."
EDIT: Not sure if it's related, but in 2015 Reddit removed it's canary [2].
[1]: https://www.theregister.co.uk/2014/09/20/apples_warrant_cana...
[2]: https://old.reddit.com/r/announcements/comments/4cqyia/for_y...
Free media is nice, but articles are written by normal people. It’d be silly to trust whatever you can read
> I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here.
https://www.bloomberg.com/view/articles/2018-10-04/computer-...
The US Government could easily find it in its self-interest to make a deal, involving the SEC, on a national security basis, to allow Amazon and Apple to go the denial route with the US Government giving them guarantees regarding fallout.
If it's worse than Bloomberg has reported, it would be highly desirable by the US Government to keep the rest as quiet as possible for as long as possible, to get at as much of what China is doing as possible. They might be running a counter intel program by now that relies on something China was doing.
"The US Government could easily find it in its self-interest to make a deal, involving the SEC, on a national security basis, to allow Amazon and Apple to go the denial route with the US Government giving them guarantees regarding fallout."
Can you point out a legal mechanism for this to be possible?
I've been in such a situation myself where I was in the room during what LOOKED like a DDOS by Akamai 10 minutes after we got off the phone with them to turn down their CDN services.
In much the same way as Apple is refuting this claim, after a few weeks of internal debate above our pay grade we decided we didn't see it and it didn't exist and therefore it didn't.
For a national security-related issue, you might just include the minimum number of people that need to know, which would naturally not include your PR team. Then when an article like this comes out, the PR team responds in exactly the way they would if it were an outright fabrication, which is what the goal would be.
> Finally, in response to questions we have received from other news organizations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.
1) Big company partners with distributor. 2) Distributor has security issues. 3) Gov is already aware of security issues, says nothing. 4) Big company becomes aware of issues. 5)Gov steps in and pitches a deal: i)Both big company and distributor must deny. ii)In return, gov gets to: iii)Preserve any existing contracts iv)Protect the big company and distributor, with any legal, trade, or commercial benefits
Additionally, Amazon and Apple are two of the still allowed to compete western companies in China that they haven't booted out of their markets, it could be that Amazon and Apple also want to retain the Chinese market and don't want to appear like they are helping while they have the cover of the NSL.
Amazon and Apple are caught in a tough spot and the hardware issue/espionage device is definitely true. It is probably just that the people that know in the company aren't allowed to say and the PR team is not part of that 'need to know' group. Either way, trust of Chinese manufacturing is over, which hits Apple very hard in both market and domestic trust.
What's stopping someone to sue after buying stock under the assumption that the companies are sound? I assume they have a duty to the shareholders.
edit: spelling.
On the other hand, Apple and Amazon have huge reasons not to blatantly lie about this. The plaintiffs bar would be all over both companies for material false and misleading statements if this turned out to be true (see also Elon's recent experiences with the SEC).
Apple and Amazon are certainly incentivized to not admit bad facts, to spin facts, to issue misleading unclear statements that read as a denial but are not, etc. But I really don't think the legal teams at either company would let executives get away with issuing such full-throated and clear denials if they were untrue.
This is so typical of Gruber as an evangelist.
There's no way that Apple would remain silent on this even if they KNEW it were true. The only possible move is denial.
Silence is validation or uncertainty, a statement of ambiguity will tank the stock and reputation as experts, recognition of even partial truth could possibly destroy their supply lines overnight.
I honestly think the corporate denials here need to be outright ignored because they have so much to lose. A story of this magnitude is basically like pointing a gun to someone's head and asking them for permission to pull the trigger.
What incentive do you see for Bloomberg to report a major story and lie!? Why would they do that?
I think at this point it's more believable the story is true, because Bloomberg is the most credible participant at this point... they're anonymous sources, but as long as Bloomberg did their due diligence as journalists, they've validated their (numerous) sources as credible.
...and like you said, all these corporations have MASSIVE amounts to lose. Just check out what Trend Micro's stock is doing today.
just like all of their other stories on Apple products/etc that have been wrong - page clicks and ad revenue. Writing about Apple gets you both. the WSJ is in the same boat. Quoting vague "sources briefed on the matter" is just mysterious enough to keep the reader on the hook for more. I haven't trusted Bloomberg to report anything accurate about Apple for years now.
As a few people pointed out in the other thread, didn't they pretty explicitly deny they were involved with PRISM?
Apple knew nothing about anything they were told was called PRISM, and they happily acknowledged the existence of APIs to service warrants.
It was presented as one of many things Amazon does to keep AWS safe at an AWS day I attended or something.
https://www.reddit.com/r/sysadmin/comments/9layb7/from_bloom...
I did a penetration test and security assessment for a major electronics manufacturer
whose parts are likely in every smartphone and laptop. I identified almost certain compromise
by the Chinese government with full access to modify the manufacturing specs using the
access paths I identified.
They chose to bury my findings as it would cause a huge stock hit. Sadly, NDA.
I'm not surprised in the slightest.So tell us what the part is pretending to be and how exactly it it was wired (and what it was connected to - is this another Intel ME backdoor?)
While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue. Every major corporation in today’s security climate is constantly responding to threats and evolving their security posture. As part of that effort we are in regular contact with a variety of vendors, industry partners and government agencies sharing information on threats, best practices and new tools. This is standard practice in the industry today. However, we have not been in contact with any government agency regarding the issues you raised.
Furthermore, Supermicro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.
https://www.cnbc.com/2018/08/30/google-titan-made-by-chinese...
'original design' is hard to verify without help from SuperMicro.
> Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community.
... especially for Amazon. Unless Bloomberg claims here that Amazon got parts of this SKU and compared them to newer parts of the SKU and found differences?
Is it legit to revise your design in terms of changes to passives without rev'ing the part and notifying downstream supply chain. Could the grain-of-rice 'microchip' be a different or new resistor/cap? Could it be logic masquerading as a passive?
> In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips.
Like a passive in a blind/buried via?
Anyways, for everyone who claims "it can't be done", "this is implausible" -- you're probably being a little naive. US intelligence agencies do shipment interdiction and adulterate products for this purpose, [1] why couldn't/shouldn't China do the same?
[1] https://www.theguardian.com/books/2014/may/12/glenn-greenwal...
I have no idea how things actually work in companies like Apple or Amazon, but would it be "normal" (at their size/scale and given their surely advanced knowledge in technology) to have inspections on the hardware they use (inspections at hardware level of the kind capable of showing these modifications)?
I mean do they routinely do these checks?
Or, assuming the rest of the article is true, someone somewhere found out about the chip and had a quiet word with a few select customers of Supermicro (but decided to otherwise keep mum about the whole thing). Which would go some way to explain the company denials, because the truth might be a bit embarrassing —or gagged by a three-letter agency.
Boards are shipping with way too much remote access capability. It's not like you need to look at system busses via the network. You're not going to debug a broken board remotely, you're going to turn it off and replace it. Now that this is an identified problem, it's time to put IPMI and its ilk back in its cage.
When it is turned off? A magical chip that works without electricity is much more valuable than any data which could be exfiltrated from servers. A chip that works without power changes the world more than the iPhone... in the iPhone Age
If this story is true, it will be an escalation clearly and willfully by the PRC from mere state-sanctioned economic espionage to an act of war against the US.
This reaction also seems reasonable if Bloomberg was telling the true.
You can't make this up, at least not in Bloomberg, that too in a front page or cover page write up. Truth could be somewhere in between and I am sure we will have more information in the coming weeks.
So what happens to the reporter then?
The difference in this case is definitely scale.
https://www.digikey.jp/product-detail/en/tdk-corporation/HHM...
The sole solution IMVHO is IMPOSE open hardware and free software by law. We simply can't have our society "nervous system" run on black boxes. It doesn't matter if the rouge in charge is China, USA, a specific vendor or someone else. Our banks, our states, our hospitals, ... relay on such connected black boxes. We also need to re-transfer knowledge from big corporate to PUBLIC, well founded, universities to AVOID dangerous evolution paths like actual IT evolution.
In both situations it's difficult to identify who is telling the truth and is probably impossible to know the truth from the outside looking in.
In both cases I've taken a "who has incentive to lie?" view on the allegations and denials, now this does not mean my hunch is correct at all, but it seems way more likely to me that it's in the best interest of Apple and Amazon to deny this story strongly as it makes both parties look bad and the investigation may be ongoing. I don't believe that Bloomberg made this story up but its understandable to question that maybe their sourcing was iffy, based on Bloomberg's history and the level of sourcing that they have cited I'm leaning towards believing the story as accurate.
For example, Dr. Ford stepped forward publicly, in full knowledge that a massive machine would swing into action to besmirch and defame her, whereas the allegations about espionage were made in confidence. The six(?) individuals making these allegations are risking nothing at this point.
Also, Dr. Ford's allegations reach back into time and are difficult to conclusively prove,* while this is a situation where physical evidence does exist somewhere. It may not be presented to us at the moment, but if the allegations are true, there were physical motherboards that could have been examined to demonstrate the exploits.
What the two have most in common is that we the public are unlikely to be provided with a full and transparent investigation. The FBI's "investigation" into Dr. Ford's allegations did not involve speaking to her or others who could corroborate aspects of her story. Others from Yale approached the FBI and did not get interviewed.
The same is probably true of this story. Even if the allegations are true, for diplomatic or other strategic reasons, the government is unlikely to shine a spotlight on the details.
* Note that "Conclusively prove," is not the same thing as "Obtain enough confidence based on other reports to make a decision about job fitness."
I guess my basic premise is that I'm more liable to believe that Bloomberg is operating in good faith in reporting this story. Now that does not mean that their sources may not be credible, but I'm more likely to believe they published this story in good faith rather than Amazon and Apple denying the story in good faith.
The credibility of their unnamed sources, their motives, and so forth, this is all unclear at this point. It may turn out that they are courageous patriots blowing the whistle on something very, very big. One can imagine a cover-up at the highest level, with these six individuals risking their careers and possibly their lives to reveal the truth.
Or one can imagine that there are some trade negotiations coming up, and a coördinated effort to plant a story so that politicians can take credit for swooping in and enacting regulations around the security of technology manufactured outside of the USA.
Who knows? I don't.
I was originally siding with Apple since they flat-out denied that they ever found a planted chip(and didn't waffle with a 'no comment' or something), but re-reading the statement seems like they may be weaseling out with their phrasing "Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server." Bloomberg claimed that Apple found suspicious chips on the motherboard, which may have not been considered a "server" at that point.
[1] https://www.cnet.com/news/nsa-surveillance-retrospective-at-...
Your article itself says that the initial reporting was incorrect and links to https://www.cnet.com/news/no-evidence-of-nsas-direct-access-... which correctly explains how PRISM is an NSA system that processes court ordered electronic wiretaps obtained by the FBI.
https://en.wikipedia.org/wiki/PRISM_(surveillance_program)#C...
I mean, what would the impact of them committing securities fraud be? Who would go after them and how much would the ensuing lawsuit cost them? The government isn’t gonna go after them if they asked them to keep it quiet.
Probably significantly less than telling their customers they’ve known about and hidden the fact that their systems were compromised for years (even if it was at the behest of the government).