Bloomberg’s chip story reveals murky world of national security reporting | Hacker News Reader

Bloomberg’s chip story reveals murky world of national security reporting

techcrunch.com

160 pointsjonbaer77 comments

Everyone's saying "remember PRISM!" but that's not how I remember it. PRISM (which now even has a wikipedia page[0]) was an interface for the NSA to browse their legally obtained FISA data from these companies. Now, I hate the FISA laws in general, but I don't think it comes as any surprise that the tech companies were following the law.

I remember the furor on HN at the time, and to my recollection a lot of the allegations were about backdoors for the NSA into their data and such, and that's what the companies denied.

Looking at Google's statement[1] of the time, I'm not sure I can find any fault with it?

To me, the big revelations from Snowden were about the NSA capturing all data on the internet backbone, and tapping unencrypted links inside Google's network without their knowledge.

[0] https://en.wikipedia.org/wiki/PRISM_(surveillance_program)

[1] https://googleblog.blogspot.com/2013/06/what.html

And compromising hardware in transit, which is basically the same thing the Chinese have done with Supermicro - just one step further upstream.

It’s a bad world, everyone is at it, nobody can be trusted blindly.

But both "Apple and Amazon are denying a report claiming spy chips from China were found in hardware they use. According to a report by Bloomberg, tiny microchips were found on motherboards of servers assembled by the San Jose company Super Micro Computer.

An official cited anonymously by Bloomberg said the supply chain-level breach affected almost 30 companies, including Amazon Web Services and Apple.

In a statement published Thursday, Apple denied the Bloomberg report, claiming malicious chips were never uncovered.

"We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed," read Apple's statement.

Amazon called the report "erroneous" in a blog post published Thursday. "At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in Super Micro motherboards in any Elemental or Amazon systems," the company said.

In a statement released Thursday, Bloomberg News said the story required more than a year of reporting and more than 100 interviews. They also said 17 individual sources confirmed the manipulated hardware."

FISA et al are unconstitutional. You're doing nobody any favors by referring to it as legitimate.

They never said that they're legitimate, simply that they're legal. Until it's overturned by a court or legislation, that remains absolutely true.

If you look at Apple denial (https://www.apple.com/newsroom/2018/10/what-businessweek-got...) they make a curious statement:

"Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple."

Compare this to statement issued by Apple in 2017 when queried about the 2016 story:

"Apple is deeply committed to protecting the privacy and security of our customers and the data we store. We are constantly monitoring for any attacks on our systems, working closely with vendors and regularly checking equipment for malware. We’re not aware of any data being transmitted to an unauthorized party nor was any infected firmware found on the servers purchased from this vendor."

(taken from https://arstechnica.com/information-technology/2017/02/apple...)

While their 2017 denial was technically correct (it was an infected driver and not infected firmware) it's still a serious red flag on their credibility on these matters.

There's a whole discussion of that at https://news.ycombinator.com/item?id=18145815 .

> While their 2017 denial was technically correct

So if you assume that their current denial is technically correct, what loop hole is there in it? Because they seem to have covered all the bases.

Was discussed in previous thread. They make very narrow statements: "Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server." This holds true if it was found by a 3rd party.

"Apple never had any contact with the FBI or any other agency about such an incident." This holds true if private 3rd party was handling the incident.

"We are not aware of any investigation by the FBI, nor are our contacts in law enforcement." This is meaningless it might not be FBI them not being aware doesn't mean there is no investigation and so on.

The Bloomberg article claimed Apple employees found the chip. So even if what you're saying is true, the article is at least wrong about how it was discovered, and that casts the rest of it into doubt as well. Also my recollection is the article claimed Apple was talking to the FBI, not Apple was contracting with a 3rd party who talked to the FBI. Similarly regardless of whether Apple was talking to the FBI directly or was going through a 3rd party, both constitute Apple "being aware" of an FBI investigation. So this only works if Apple contracted a private 3rd party to audit their servers, the 3rd party found the chips, didn't tell Apple, and talked to the FBI all without informing Apple of the issue. This strikes me as extremely implausible.

Also, I don't think I'd accept the claim that "Apple has never found" is telling the truth if it was a 3rd party that found it. Because if a 3rd party informs Apple, that report right there constitutes Apple finding it.

Wouldn't this fall under the remit (and expertise) of the NSA? I thought that the FBI were like Interpol - i.e. federal-level cops. The NSA being the technical spooks (and the CIA being the meat-based spooks).

I believe there's been a bit of a shake up since 90s - at the time secret service got the hacker beat, because: wirefraud. But also the FBI, because: felony crossing state borders.

The nsa were signals intelligence first, but their civilian mandate had (has) to do with protecting national interests in the "signaling" world (ie: the Internet etc). Arguably they were never very good at that... ("Snowden", "crypto backdoor"...).

But I believe "cyberspace" is now accepted as an actual thing, and so falls naturally under the FBI (cross border, spying on us soil) and police ("crime").

It could be trivially done through technicalities.

For example if the exploit was found on the motherboard prior to it being deployed in servers that would be consistent with Apple's denial (not suggesting that's what happened here, but more as an illustration of how Apple being specific actually leaves more wiggle room than broad statements).

It's also possible that amidst the trade war China vs US, a few people in the US fed a business-oriented outlet about Chinese spying. You can add micro chips in a few Supermicro servers and Bloomberg journalists would eat the story like candy. This would dramatically decrease electronics imports from China, and putting even more pressure for trade talks. See the Vice President's speech yesterday .

I could be wrong. But we had previously paranoia about Japanese corporate spying 40 years ago.

> It's also possible that amidst the trade war China vs US, a few people in the US fed a business-oriented outlet about Chinese spying.

On the other hand, "vehemently deny this or our business relationship will be soured" is exactly what you would expect from China on this. It's not as if censorship isn't in their playbook or putting a lid on this isn't in their interest.

It would be very disappointing to see US companies cowed by something like that, but it's not as if US companies knuckling under to China's censorship requirements is without precedent.

For anyone wanting a citation, China recently threatened to ban airlines from flying to China unless their global websites (including outside of China) were changed to list Taiwan, Hong Kong & Macau as part of China:

http://www.abc.net.au/news/2018-06-04/qantas-to-refer-to-tai...

Several American airlines (Delta, American, United, Hawaiian) agreed to comply:

https://www.bloomberg.com/news/articles/2018-07-24/u-s-airli...

I agree with this thinking. It is also possible that competitors to multi-billion contract being awarded to AWS by DoD wants an argument to distributing the contract among many cloud suppliers to reduce risk. Bloomberg has worked with this story for a year - is that a man-year or a calendar year? If the latter, both trade-war and the DoD contract are less probable.

This is exactly what i was thinking as well. And I am aware i could be wrong too but it is also a possibility we should not discount

bingo

This makes for interesting reading alongside:

* The Register's analysis https://www.theregister.co.uk/2018/10/04/supermicro_bloomber... (https://news.ycombinator.com/item?id=18146307)

* Joe FitzPatrick's analysis https://securinghardware.com/articles/hardware-implants/ (https://news.ycombinator.com/item?id=18144538)

Once upon a time, a university friend of mine spent his summers working for the Australian public service. His response to the rumours about Eschalon, before the Australian spooks confirmed them [1], was: "They're capable of doing that. So of course they're bloody doing it." That stayed with me.

Maybe it doesn't matter if these chips were implanted in the particular boards that they're alleged to have been in. No one is suggesting that this would be technically infeasible, so you can bet the PLA has planted them somewhere. I would, if I was defending China against Trump.

[1] That drunken Christmas party excepted.

> Yet, to Apple — and Amazon and other companies implicated by the report — they too might also be in the dark. Assuming there was an active espionage investigation into the alleged actions of a foreign government, you can bet that only a handful of people at these companies will be even cursorily aware of the situation. U.S. surveillance and counter-espionage laws restrict who can be told about classified information or investigations.

I think this may be key in this case, which would give plausible deniability to Apple and Amazon. Conversely, I'm also not fully convinced by this argument, I think it applies more easily and more often when companies are directly subpoanaed by authorities not when there are the initiator. In the case of this kind of breach if one engineer find this issue I would think it would report to senior management first, before contacting the authorities. Also Apple directly stated there are not constrained by any gag order, which leave only one possibility if they genuinely think what they say is true: could it be an unkown unknown?

I’d think the denial press releases were cleared all the way up to the respective CEOs and even if there were such sensitive conversations with the FBI that the CEOs weren’t party to the details, they’d still know of their existence.

I’d be surprised if Justice Department guidelines allow completely going behind the back of the executives of a domestic public company, at least unless they are suspected.

I can't see how Amazon could cooperate with the FBI investigation without informing the FBI-veteran CISO

All I can say is that if these chips don't _actually_ show up soon and get analyzed, Bloomberg is going to have a serious black mark on their credibility.

How many days can this go on without _SOME_ report of these things after such a ball-buster story?

Indeed. They also said a major hosting company got sabotaged gear. Surely someone at said hosting company took a picture of the board, kept a board, or something. People at hosting companies aren't paid very much, so they're not incredibly loyal. Still they must have said something to friends, family, facility workers (why are you throwing away all that supermicro gear?), etc.

It doesn't make sense that no one has produced at least a picture of one. Why the need for secrecy here? It's not like the US govt made the chip, right? Right? RIGHT?

Where reporters across any topic and beat try to seek the truth, tapping information from the intelligence community is near impossible. For spies and diplomats, it’s illegal to share classified information with anyone and can be — and is — punishable by time in prison.

Even worse, feeding reporters false information is not that difficult. Given the scarcity of sources, it must be extremely difficult to get technically knowledgeable people who will corroborate these kinds of stories.

Consider who wins the most if the chip story turns out to be false: an administration hell-bent on reshoring US manufacturing capability.

Whether the story is true or part of some domestic propaganda operation, the result isn't good for the US.

Why wouldn't the US gov show one of the infected motherboards? It seems like it would be perfect proof that China is engaged in aggressive hacking. The story is in the open now. If it's true, China knows that we know. So what else is there to hide?

Maybe they were happy to use the same backdoors for themselves?

Maybe “they” were happy to have China do the dirty work and take the fall, but it was all conceived and execute by “them.”

I think its pretty clear from context that "they" refers to the intelligence agencies of the US government.

Oh there is no confusion over who we are referencing.

In this case, maybe using a software or firmware virus could be a better way? Easy to install, easy to remove, leaves no evidence if be careful.

This got me thinking, why plant a chip? I mean, it's hard to make, and will almost certainly be caught.

Hardware stays put, even very targeted viruses don't as evidenced by stuxnet.

That theory would explain a lot

Flip side is the US govt placed these chips and are deflecting the blame to China.

Because it takes weeks/months to get such things declassified through standard channels. And all the non-standard ways require efforts by elected officials. They are all busy with something else atm.

Because someone could interpret chips like this as an act of war?

Couldn't the hackers just swap a chip for a duplicate that has the extra "chip" within it. Then all you have to show is identical looking boards, and it doesn't matter if you can show the difference technically, if people see identical looking boards they'll say it's fake.

More directly relevant, why doesn't bloomberg show one of them in practice? That would really help in confirming the story. (Not that I doubt bloomberg here)

> Today’s bombshell Bloomberg story has the internet split: either the story is right, and reporters have uncovered one of the largest and jarring breaches of the U.S. tech industry by a foreign adversary… or it’s not, and a lot of people screwed up.

Imagine reading this after the Snowden leaks:

Today’s bombshell Guardian story has the internet split: either the story is right, and the biggest Silicon Valley companies are giving the NSA access to their data… or it’s not, and a lot of people screwed up.

Two different stories, about two very different situations. There’s not really much equivalence.

The report contains elements are are near impossible, and has other aspects that are very unlikely.

It has also been denied far more strongly than the Snowden leaks were.

The biggest problem is really why would you do this? Either the report is wrong in significant ways, or it’s not true at all.

The chip they show in the article, is a ceramic package which it would be really hard to embed a semiconductor in (because of the temperatures required to fire the ceramic). It looks like it probably would sit on an alternate footprint for the BCM flash. A ceramic part like that (which they say is for signal conditioning) doesn’t belong at that location anyway.

If your going to develop some weird SMD capacitor sized package for a microcontroller... why not just develop a new BCM serial flash chip embedding the same functionally? At least that way the boards would look visually similar.

So much just doesn’t make sense to me.

The Snowden leaks were denied plenty by the companies involved.

https://googleblog.blogspot.com/2013/06/what.html

And is there anything in that denial that has been shown to be technically incorrect?

My understanding is that there wasn’t involvement from google, that data was extracted without their knowledge from links between data centers.

And what about Amazon and Apple's denial will be found technically incorrect in a few years.

EDIT: to the downvoters, its easy to play this game, Apple makes a large amount of very specific denials.

    Each time, we have conducted rigorous internal
    investigations based on their inquiries and each 
    time we have found absolutely no evidence to support 
    any of them.

    Apple has never found malicious chips, “hardware 
    manipulations” or vulnerabilities purposely planted 
    in any server. Apple never had any contact with the FBI
    or any other agency about such an incident. We are not
    aware of any investigation by the FBI, nor are our 
    contacts in law enforcement.

What about investigations and law enforcement contacts via a third party (perhaps a specialized hardware security firm? )

    Our best guess is that they are confusing their story
    with a previously-reported 2016 incident in which we 
    discovered an infected driver on a single Super Micro 
    server in one of our labs.

You mean the story they explicitly denied with similar strength (and wiggle room to boot when the truth came out)

    Apple is deeply committed to protecting the privacy and
    security of our customers and the data we store. We are
    constantly monitoring for any attacks on our systems,
    working closely with vendors and regularly checking 
    equipment for malware. We’re not aware of any data being
    transmitted to an unauthorized party nor was any
    infected firmware found on the servers purchased from 
    this vendor.

https://arstechnica.com/information-technology/2017/02/apple...

These denials match up exactly with what was in the leaked documents. Greenwald and Snowden, out of incompetence, made a bunch of unsubstantiated accusations that were not supported by the documents that the companies denied.

Actually, there was a storm of denial and attacks on snowden's character in main stream media after the leaks ("high school drop out", "low level contractor" etc).

The Guardian is an established newspaper with a firm commitment to (what it sees as) the truth, and has a top-flight internal technical team which gives them credibility on this kind of topic. (Interestingly we did see an internet split after the Guardian reported vulnerabilities in WhatsApp's security, but at least in that case there was no dispute on the facts, only their interpretation).

Bloomberg is an upstart with an awkward funding model which has already faced serious, credible allegations of political interference and a lack of journalistic integrity (a major investigative story about corruption in China was allegedly spiked at a late stage by management for business reasons).

I wouldn't call them an upstart. They've been around for 23 years and produce some impressive content, but you are correct regarding the story about the families of the Chinese leaders and potential conflict of interest given Bloomberg L.P.'s business.

After the Xi story appeared, terminal sales in China slowed because government officials ordered state enterprises not to subscribe. The Bloomberg News website was also blocked on Chinese servers, and the company was unable to get visas for journalists it wanted to send to China.

In 2014, Grauer told staff at the Bloomberg Hong Kong bureau that the company's sales team had done a "heroic job" of mending relations with Chinese officials who had indicated their displeasure about publication of the Xi revelations. He also warned that if Bloomberg "were to do anything like" the Xi story again, the company would "be straight back in the shit-box."

On October 29, 2013, during a conference call, Winkler told four Bloomberg journalists in Hong Kong that the findings of their major investigation into "the hidden financial ties between one of the wealthiest men in China and the families of top Chinese leaders" would not be published. Less than a week later, a second planned article "about the children of senior Chinese officials employed by foreign banks," was also killed, according to Bloomberg employees

https://en.wikipedia.org/wiki/Bloomberg_News#China_coverage

Are we meaning the same Bloomberg News? Hasn’t it been around, at least for business news, since about 1990? If so, that seems to me to be well past the upstart stage. But maybe they rebranded or reformed and have no relation to the other name?

I perceive them as an upstart in traditional-journalism terms, certainly compared to a newspaper that was founded in 1821.

So they're lying because they've only been around for 30 years? What sort of logic is that.

They don't have the same level of history and reputation that the Guardian does. So people don't trust them as much as the Guardian. Which seems fair and reasonable.

It is shoddy journalism, or rather propaganda when anonymous is really a CIA/MOSSAD etc, as is the case with Bloomberg et al.

I find it somewhat curious that the 3rd party in Ontario, Canada that actually found was and remains unidentified.

Was that off-the-record for the story? Or delibrately omitted? It seems unlikely they are part of the intelligence community so protected in any way; they're in another country and they must be somewhat known in the DC industry if Amazon used them commercially.

I have no special knowledge about this, but Chipworks is a well-respected Ottawa (Ontario, Canada) company that has been doing silicon-level reverse engineering and teardowns for a while. It looks like they're part of TechInsights now, but seems like the right kind of crew to be doing this.

See eg. some silicon analysis on the iPhone 7 http://www.techinsights.com/about-techinsights/overview/blog...

This is pretty fascinating. The statements by Apple and Amazon on one side vs. Bloomberg are very hard to reconcile.

I'm very interested to see where this goes. I hope we get to find out who is full or crap on this. Either Bloomberg got seriously played (I'm assuming they wouldn't just make up stuff for a good story or report based on sources that didn't appear credible) or Apple and Amazon are lying fearlessly and in great detail. This doesn't seem like the prism situation where it was pretty easy to reconcile the company PR statements with the snowden leaks.

Just to throw some extra info into the mix, Australia's media [1] is reporting that the Australian Department of Defence and Bureau of Meteorology also had contracts with Supermicro. The current Australian Defence Department statement:

A Defence spokesperson said the department was "aware of recent media reporting involving the unauthorised implantation of microchips within servers, used by United States corporations, in the production of Supermicro microchips".

"Defence will continue to work with the ACSC [Australian Cyber Security Centre] to continue to monitor the situation," the spokesperson said.

And from the Bureau of Meteorology:

The Bureau of Meteorology said it does not comment on security matters.

[1] http://www.abc.net.au/news/science/2018-10-05/supermicro-mal...

Easy to reconcile them after the facts all came out maybe, who says this time will be different?

How did Bloomberg get the glitzy photographs? Were they just cooked up to have something to show in the article or did their source provide them? What legitimate investigation would have used a pencil as part of their protocol for photographing evidence?

It seems that it would only be interesting if Bloomberg was right.

People lose trust to companies like Apple or Amazon if they lie. Bloomberg though? Nope, everything would just continue as normal. And Bloomberg is not even that bad. We just got used to bad journalism.

I am skeptical of the story. Bloomberg should have at least had pictures of the chips on a board, something from reality. Instead they had illustrations. These were commercially available products, they should have independently found a smoking gun board sample with such chips and analyzed it, and not take their sources' word for it. This seems like a planted story to me.

post has been deleted sometime after January this year: http://web.archive.org/web/20180126235643/https://blog.cari....

I think it's time Bloomberg either puts up or shuts up. It's that simple.

Either they can show an xray of a motherboard showing the chip, and can further explain how it exfiltrates data, or their story is rumour and bullshit, and they should be culpable for Supermicro's stock drop.

It's just that simple, and I'm calling them on it.

It might not be that simple. If the story was done properly, Bloomberg received information from a source that they were able to verify using other sources. Maybe they saw parts of the report they talk about, but likely they just talked to sources. Bloomberg likely cannot reveal anymore information without revealing sources. Their options are to stand by their story, or retract it.

I'm waiting for another news outlet to bring more information. Additional sources will come forward and more reporting on this story will only get us closer to the truth.

I disagree. Bloomberg knew what kind of financial effect the story would have on at least Supermicro, and if they can't back up the story with actual, provable facts, then they are culpab!e. They know this.

So let's have some facts. Like an x-ray.

FYI: "Britain’s national cyber security agency said on Friday it had no reason to doubt the assessments made by Apple and Amazon that refuted a Bloomberg story that their systems contained malicious computer chips inserted by Chinese intelligence. [...]"

https://www.reuters.com/article/us-china-cyber-britain/uk-cy...

It's a fairly non-committal statement below a Reuters headline that is not borne out by the actual article. And it is not alone.

* https://news.ycombinator.com/item?id=18148811

If the story is true, show me some of these spy chips. It should not be hard to indicate the exact location and allow anyone with supermicro motherboards find them.

You haven’t read the article properly. Just specific batches for targeted Supermicro customers were infected.

I don't understand why people are discounting the fact that the said companies wants to do business in China and you can't do that by accepting the Chinese state is trying to hack their companies.

IMHO, this is a very big aspect and companies lie all the time. Even if this came out to be false, they don't get as much heat as they would get now.

What's the point of this article exactly? To make an excuse from the Bloomberg reporters? Cause all clues point to the fact they made that story up.

https://twitter.com/TubeTimeUS/status/1047979340477083648

apple is deeply commited to discouraging people from reversing and inspecting their products, just like oracle.

Hey Bloomberg, why don’t you show the world one server motherboard with the Big Hack Chinese Chip on it?

Why is it we believe these companies statements?

Is the PR of fortune 500 companies bound to tell the truth always?

I'm a capitalist, but I cannot deny that it is in the best interest of companies to hide and deny negative news.

It's not in their interest to issue a strong denial if they know the story is going to be confirmed later. It just makes them look worse than if they'd issued some generic language about how much they care about security. So either they genuinely believe the story is false, or they believe it's impossible to prove, and given the scope of the story I'm not sure there's a difference.

Lying to the shareholders is criminal (maybe civil offense, IANAL) if you're a public company.

Reply on news.ycombinator.com