Hacker News Reader

Comment by Jerry2 | Hacker News Reader

Both Apple and Amazon have released VERY STRONG denial statements that bring the whole Bloomberg narrative into question. It's also convenient that no one has yet been able to verify or find any of these mysterious Chinese chips on any of the Supermicro servers in the wild.

So what is the real story here? Did Bloomberg reporters deliberately deceive everyone or were they deceived by the US IC ("intelligence community") as a way to scare technology companies from doing business in China?

Someone at the SEC should scrutinize SMCI shorts at the very least.

View on HN

Remember when Clapper gave the "least untruthful answer possible" about domestic bulk collection? [0]

It's naive to think Apple and Amazon couldn't lie in response to the article, if the intelligence was highly-classified. They may be under extremely strict gag orders (e.g. "give no response whatsoever, including silence, other than denial") and protected by promises of indemnity, as telcos were in the wake of the NSA disclosures.

Bloomberg is exquisitely specific and detailed about the chip, including photos and placement (on the baseboard management bridge, disguised as a shielding element.) There's two possibilities, then: someone cajoled dozens of officials across several companies to lie to Bloomberg, or the tech companies have been cajoled into lying.

Either way, the proof is in the pudding: every hardware pentesting shop will be going after these boards like they're looking for golden tickets. Either we'll get our die shot, or Bloomberg's getting their pants sued off.

0. https://m.youtube.com/watch?v=Jkb5FKlETqY

I think there is no way they would claim they are not under any gag order if they were forced to never confirm such an order, if there also is the option of not addressing it at all.

  I think there is no way they would claim they are not
  under any gag order if they were forced to never confirm
  such an order

I've always thought such denials are pointless.

You think everyone at Google knows every secret gag order they're under? Of course not, that would get leaked within minutes.

If I was the NSA I'd find a "patriotic" mid-level or junior employee with the power I wanted to subvert, and give them a gagging order that stopped them telling their boss.

That way the Google PR can honestly say that the CEO and legal team haven't seen or heard of a gagging order.

I think it would be odd then if no one else in the company is aware of the issue that those mid/junior level employees would then leak the information to Bloomberg during an active investigation.

> ...every hardware pentesting shop will be going after these boards like they're looking for golden tickets.

The way the original story was written, it suggested that four subcontractors were identified, and almost 30 targets selected, with the implied suggestion that either the boards were custom special order boards, or destined for a specific lot order made by a customer. If true, then it is unlikely these boards will be found in the wild except by slipping through those almost 30 targets into the used market.

Supermicro has about 600 board SKUs, so finding these needles in the haystack is likely more feasible by approaching data centers offering to help find the boards for free, in return for allowing the pentesting firms to take physical possession of such found boards.

The story did reveal that the original "tell" that gave away the chip was odd but not obviously malicious at first glance network traffic, and that the suspected intent was implementing an Advanced Persistent Threat model. The article also mentioned the chips were connected with the BMC, but it wasn't specified in the article whether or not the chips got out onto the Net.

So finding the boards in the wild probably will focus upon finding them in the same manner, over the network. Power down the server, let the BMC stay powered on, and watch for unwarranted network activity. Or boot a Linux on a stick that deliberately does as little as possible and premises networking allows it to do just enough routing out to the Net to capture traces of what unauthorized network traffic is trying to do, and watch for unwarranted network activity, in case the chip design is clever, and hides its activity until it detects the mainboard is already running before trying to inject its network payloads onto the mainboard's network interfaces.

What are others' thoughts on how to find these "golden ticket" boards?

That's the interesting bit. Having done so much to hide the exploit your most important aim is to hide it's presence. Anything that would just "connect to a Chinese server" would be discovered immediately. If the Bloomberg story is true, the network traffic scheme must have been extremely sophisticated to fool so many network security specialists at top companies for such a long time.

This, or the story is false, purposely or not.

Most places I know of isolate their OOB management network, requiring a vpn or jumpbox to access it. However, if someone did let their OOB network full outbound access, I could see this slipping through. I could imagine that simply going to a CDN or cloud provider like AWS/Cloudfront/cloudflare/akamai with a dns lookup along the lines of updates.supermicro.cdn-front.com wouldn't be too suspicious. At that point, you'd be looking for dns lookups and not firewall hits.

If you are blocking outbound, I could still this going unnoticed if you're not actively reviewing denials.

But, if you are properly watching dns lookups from OOB and it's anything other than necessary services (ntp, ldap, syslog), then this would get picked up pretty quickly.

Sounds like many things would have to go right in the defender's court. Optimism is not a good defense strategy :)

Heh. One colleague has "Hope is not a valid deployment strategy" as a signature.

Most places might be blocking this type of activity by default. For most of our security audits, it's just assumed that the SM IPMI or Dell idrac is vulnerable to one exploit or another. We mitigate that by controlling the traffic. I feel this is common practice in most places that understand vlans and firewalls.

However, while blocking is easy, being aware of something like this is on another level altogether. Unicorn jumping over a rainbow level rare. You really have to be logging outbound attempts and dns lookups. Where I work, there is a full security team and they are at an insane level where they log the allowed traffic. One told me that the allowed traffic is more interesting than the denied traffic. Denied just tells them what we anticipated, while active helps them establish a pattern and look for deviations.

That's my point. Security people were analyzing network traffic for decades trying to spot something that doesn't fit, host-wise, pattern-wise or even packet-wise (see The Museum of Broken Packets[0], for example). And someone managed to somehow hide all this traffic from security experts working for Amazon and Apple, for months or years? I'm very curious to see how.

[0] http://lcamtuf.coredump.cx/mobp/

I think you misread - "Unicorn jumping over a rainbow level rare" was about catching it, not missing it.

> However, if someone did let their OOB network full outbound access, I could see this slipping through.

Quite, especially in small networks and inline ilos, entirely possible that people plug the ilo (ipmi etc) to a more open network. Sure, nothing in, but no block on stuff going out.

I'd appreciate a detailed explanation of the steps needed to get from putting this chip on a motherboard to actionable intelligence useful to a hostile nation state.

If we're talking about being able to make changes at the OS level, surely those should be relatively easy to spot?

If we're talking about copying packets, wouldn't that be useless under most circumstances?

If we're talking about doing whatever on a mass scale, surely most of the data would be junk and a huge and very clever backend would be needed to sift out the useful elements?

Presumably none of this is impossible, but I haven't yet seen a good description of how it's all supposed to fit together.

Something like triggering a memory-read of a specific address the moment you see a TLS diffie helman exchange on the wire? The idea being that you can recover the session key, and thus break the encryption retroactively.

That requires more than just access to the network, but it is rather simple.

With just access to the network, perhaps one could do spoofed DNS responses that are never seen on the network? A very simple '1% of the time, set the gmail.com A record to the chinese gmail IP' might be enough.

One could create mayhem by sending false ARP or DHCP responses, but that is only mayhem. Perhaps if it is externally triggerable it is useful offensively as DoS.

That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards?

Or are these boards commercial-of-the-shelf things that are mass produced?

In the latter case these boards can just end up anywhere, in the first case not so much. As a result using commodity boards would provide some degree of protection against such an attack as slipping a manipulated one into a specific target's servers would be a game of chance. Also the risk that the attack is discovered would be much higher.

> That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards?

Why not go read the story? It addresses this question and many others HNers are asking now. It's actually a fairly detailed story, although annoyingly short on technical detail (which could have been because their sources refused to go into much detail).

I did read it, so if it was mentioned I missed the part where Bloomberg stated if it was targeted at specific orders and thus customers or if it was seeded into the supply chain to see what sticks.

So, if you know the answer, why not just providing the information? ;-)

I think the Command and Control traffic will be much harder to spot. In fact it might be dormant until an operative is on the same network and wakes it up.

> Either way, the proof is in the pudding: every hardware pentesting shop will be going after these boards like they're looking for golden tickets. Either we'll get our die shot, or Bloomberg's getting their pants sued off.

From the story, this seems to have happened several years ago.

I wouldn't be surprised if house cleaning had been done for some time.

If the story is true then I doubt that the Chinese would have flooded the market with these boards. It must have been relatively targeted.

The proof of the pudding is in the eating. That makes sense. I ate the pudding it tasted good it proved to me its deliciousness. There is no proof in a pudding that is not eaten.

You clearly not an Agatha Christie Fan

This isn't how gag orders (or, in the case of online services, NSLs) work. A gag order can compel you not to share data, but they cannot compel you to lie, let alone lie elaborately, as Apple and Amazon at this point would have to be for their denials to be false.

If Apple is lying here, they're lying because they want to, not because something is forcing them to. I believe them.

"Did you receive an NSL?"

You have been compel to lie, by the government, to answer "NO" if you've received one.

If you can be compelled to lie that you have received an NSL then I do not see why you can't be forced to lie more.

We already compelled speech, why not more?

I thought that you could simply not answer the NSL question (hence warrant canaries)

Iirc that's only in theory. I don't think there has been any precedent set. But I always see people say that a warrant canary is a false sense of security.

> This isn't how gag orders (or, in the case of online services, NSLs) work. A gag order can compel you not to share data, but they cannot compel you to lie, let alone lie elaborately

Is there case-law to support this? I've heard the argument often, but I don't believe it has yet been tested.

It’s a well established doctrine that this is a direct consequence of First Amedment: https://mtsu.edu/first-amendment/article/933/compelled-speec...

> Bloomberg is exquisitely specific and detailed about the chip, including photos and placement (on the baseboard management bridge, disguised as a shielding element.)

They are very specific, yes. They show pictures of the alleged chips that are allegedly disguised, yes. They describe placement, yes.

However, photographic or video evidence of these chips on SuperMicro boards is conspicuously absent. The only visual of placement is an illustration, not a photo. I’m not suggesting Bloomberg is wrong. I find this story fascinating and incredible (if true). But I noticed there were no photos/video of the chips on boards.

One thing that sticks out to me is that they are fairly persistent on using "Apple" rather than something like "Apple and its security partners and subcontractors", which would eliminate most of the potential for doubt.

The last paragraph of the press release on this hacker news post:

"Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations."

So you're basically implying that Apple is colluding with the US government to actively lie to discredit this story.

The funny part is that, based on your tone, you can't imagine a case where: 1) that's possible and 2) Apple may not be a willing colluding party.

I can't imagine a case where it's within the US's power to force Apple to falsely deny a story for them, but not within their power to feed a fake story to Bloomberg.

The reason Bloomberg is so sure about this is because chips/'infected' Supermicro boards were originally found at Bloomberg. They noticed odd web traffic coming from a server, took a look, found nothing, looked closer, and finally found a hardware exploit.

What you're seeing in the Bloomberg piece is a bunch of half-truths backed by soild data. It is a BMC exploit, and they are doing it through the BMC EPROM, and even the position of the 'exploit' in the article's graphics is accurate. This is obfuscated because of an ongoing investigation. It's real, but there's purposeful misinformation, combined with a journalistic game of telephone or chinese whispers.

It's also about the worst possible way to drop and 0-day, but whatever.

This also illustrates why hardware implants do not work as a mass infiltration tool. The idea is not getting caught, otherwise risking the whole operation. Implementing a threat like this is counterproductive.

Intel ME seems pretty successful though.

This amazes me. Everyone freaks about the Chinese doing it, whereas we're surrounded by millions of computers doing the same thing - or at least able to - without us even knowing or being able to block it.

I suppose a big difference is that Intel ME is on the spec sheet, is known about, and is sold as a feature for sysadmins. It's not cool that it's not optional, but it can be switched off.

Why would Apple and Amazon release such vehement denials, though? If it were an ongoing investigation, wouldn't they use more hedging and obfuscatory language? They could just as easily say "we're not aware of anything like this, but we take all allegations of this nature seriously and are looking into it". The whole situation is just odd.

Obviously to quiet their contractor and client fears. Shareholders know that replacing the hardware would be a major cost and PR damage intense.

It's still an odd strategic move though. They could get caught in the lie, in which case they would both be exposed to pretty serious legal and financial trouble.

Replacing the hardware would be cheap compared to the hit they could take getting caught in such an outright lie. Public companies have obligations to their shareholders not to lie about things like this. They're opening themselves up to lawsuits.

Their reputations could take a huge hit from this story alone, so a strong denial is the first defense. If they are unaware of any investigation, as they claim, then that makes even more sense.

The part I don't understand is why Apple would take such a huge PR hit from the story. I way I read it is that they found out about the problem, diagnosed it and replaced all servers all within a very short time. A careful writeup about what happened, what kind of information that might have leaked, and what they do to avoid it in the future would to me be a net PR win at this point. The next question would be how all other customers has acted.

If Apple were to agree with and state that China is spying on them, then they might face repercussions from the Chinese government on their extensive business interests in China. They make all the stuff there. This is monetary threat to Apple, so lying is within character for a corporation to preserve their business.

If the story were true Apple cannot lie about it publicly like this; the SEC would have their heads.

They would have to hedge, "we are investigating Bloomberg's allegations and take security seriously."

I wonder why people on HN take these arguments seriously.

You either anger govt that is authoritarian & capable of confiscating your entire production (basically ending your current business)

You make some half-truth statement (not necessary to be lying, but that does not change the point), and - if everything will be revealed - one agency of democratic govt will be pissed and slap you with big fine.

Basically this is big enough to make SEC meaningless.

The idea that the Chinese gov't would seize Apple's "entire production" is up there with the possibility that the US invades China. It isn't going to happen. There are just too many consequences, not least of which is the withdrawal of most US companies from China.

The SEC on the other hand literally just got done fining Elon Musk $20m for a tweet. Fines for something like this could be in the hundreds of millions, that's not something you risk when you could just say "we are investigating the claims and take them seriously."

SEC fined him over something so much easier to verify that I wonder why is everybody bringing it up? Why do you think truthiness of Apple'a / Amazon's statements will be obvious in next year or two? Why do you think that accussation like "China is hacking our HW" from Apple would not escalate situation dramatically? (implicating they have to deny as hard as possible)

It's such obvious damage control in shit-has-publicly-hit-the-fan situation ... Why do you think SEC's fines (lol $20M, even for Tesla/Musk it's not a big deal, Apple would have to be hit with 100times that to even notice it) are even relevant now?

And "It isn't going to happen." could have been said about hardware attacks on USA from China last week. Situation has changed quite a bit, China is either starting to abuse its hw monopoly or someone is doing A LOT to prevent such scenario, so the stakes are too high for simple SEC fines to be worth discussing.

Lying publicly and on the record about something that should have been disclosed in their financials as a risk is not something the SEC would take kindly to.

Publicly traded companies do not have the same freedom to lie because the truth or falsity of this claim has a direct impact on their stock price. Something we just saw with Musk less than a week ago.

I certainly understand why they wouldn't want people to know about it. But lying outright in a public statement in a way that is direct and clear could expose them to serious legal risk. Trading PR risk for legal risk seems like a dangerous move that companies this large wouldn't take lightly.

For me, the Apple piece is not as black and white.

> "Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple."

They deny Bloomberg's specifics, but basically admit that the general attack vector very much works.

That quote seems to be describing a software attack, not a hardware attack.

> "chinese whispers"

Does anyone have a better phrase for this?

A Chinese workmate called me out for using that phrase in his presence. Until then I had used the phrase since childhood without thinking that there were connotations and with nobody complaining. However, after that one fateful conversation I did see that my language could be improved. But how?

Does anyone have a concise alternative that conveys the same thing without implying anything about how Chinese people communicate in whispers, to get things wrong?

Not a native speaker: Does it have additional meaning to the reference to the children's game where everyone whispers to their neighbor, passing a message along? If not, isn't that also known as the "telephone game"?

I was going to reply "I don't know, it's all Greek to me" but then decided that was not a good idea. So, [0] says

> Chinese whispers is the British term for what is known as the telephone game in the United States

and then lists several other names for the same game. I've also met people who when discussing confusing messages say "send three and fourpence, we're going to a dance", which is a result of 'Chinese whispers' being applied to the input "send reinforcements, we're going to advance". The reference to the pre-decimal coinage ('three and fourpence', i.e. three shillings and four pence, approx £0.17) in this old phrase shows how long the concept has been around in British English (we decimalised in 1971).

[0] https://en.wikipedia.org/wiki/Chinese_whispers

In France it is the "Arab telephone" !

I had never even heard the phrase until this thread. A quick google/wikipedia shows me it's the British term for what we in the US call "telephone". I guess I learned something new today.

https://en.wikipedia.org/wiki/Chinese_whispers#Etymology

> As the game is popular among children worldwide, it is also known under various other names depending on locality, such as Russian scandal,[3] whisper down the lane, broken telephone, operator, grapevine, gossip, don't drink the milk, secret message, the messenger game, and pass the message among others.[1] In France, it is called téléphone arabe (Arabic telephone) or téléphone sans fil (wireless telephone).[4][better source needed] In Malaysia, this game is commonly referred to as telefon rosak, which translates to broken telephone. In the United States, the game is known under the name telephone – which in this use is never shortened to the colloquial and more common word phone.

The Telephone Game is not common parlance in British English, just American English. So a better phrase is sought than that to convey the idea.

I suspect there is no common phrase. Chinese whispers isn't common parlance in american english.

"Pass the message" or just "whispering game" seems to be the same.

To be clear, there is a very real threat of China embedding chips that do "bad things" inside chips that are sold to the US (source: I once worked on a government contract and saw such chips - they were, to my knowledge, unrelated to this specific case). I'm not sure where that points, but there are strong motivations at play.

Not the least because since Snowden we know the NSA does it.

The key is ,,inside chips''. There's no reason to do those bad things on the die using an extra chip, as it's too easy to prove.

It's certainly beginning to look like Bloomberg got played. I don't know what the motivation would be for Bloomberg to deliberately deceive readers, but I can't help notice that the article came out at roughly the same time as Mike Pence was giving a speech whose central premise was that China is meddling in american politics, and all the attention currently focused on Russian hacking should be focused on Chinese hacking.

If this story actually is false the US government has plenty of incentive to have planted it.

Did you miss the other hacking news from yesterday?

https://www.cnbc.com/2018/10/04/doj-charges-7-russian-intell...

But go read the Chinese response. It certainly looks like a non-denial to me. Pretty much says, "yeah, it was us, but US is doing the same to us".

The full statement can be found at the link below, but the take-away is pretty much this quote from the statement: "Supply chain safety in cyberspace is an issue of common concern, and China is also a victim."

https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

My general response whenever there is something like this: Look at the past stories by the reporter. Bloomberg handily lists them for you. https://www.bloomberg.com/authors/AQrv1y2ieI0/jordan-roberts...

Take a look at those stories in the strong light of being able to see them in retrospect and decide for yourself if this reporter is prone to going to press without a full understanding of the situation.

I have my opinion, but it's best for people to form their own.

hmmm... definitely a trend of reporting on hacks. And perhaps with sensationalist clickbait headlines.

But they are confirmed hacks (Equifax, FB, Cambridge Analytica).

The headlines are somewhat over-the-top, but none of those articles show examples of them deliberately making up a story.

I wouldn't read it, because they seem to repeatedly overstate the possible impact of their stories. But if you work in the space, or are the victim of an attack, this thread-level-orange style of writing may feel entirely appropriate.

It's no different than, say, insinuating that something is wrong with <x>. But instead of clearly explaining the criticism, to "let smart people figure it out for themselves". That's the laziest conspiracy sales tactic, and it exploits peoples' insecurities by tying the carrot ("I'm one of the smart ones!") to the stick of accepting whatever theory is being peddled.

In this case, there's relatively little ground for misunderstanding or exaggeration. There's either a malicious chip or there's not. I suppose in the "worst case" scenario, there could be a malicious chip, but some other government than the Chinese put it there.

However, reading the statement from the Chinese government in response to this story, which never actually denies the incident and instead complains about being victim to such hardware hacking themselves, that's not the impression I get: https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

Another option is the attacker has those companies "by the balls". They knew that the supply chain was completely compromized, and has been for a while. Admitting this, after so much time, is financial suicide in the best case. Maybe the companies are even being blackmailed by the attackers.

Another twist, maybe someone in the government does not want this to come out officially, because they would be forced to take action against China.

On the other hand, maybe this is exactly what the government wants, so they "leak" this information - be it true or false - on purpose.

In the end, the world is so messed up currently that every conspiracy theory seems to be equally plausible.

Puts these somewhat irrational Trump claims from July into context? *This is speculation/conspiracy, but if they've been looking into this for years, I could see the Trump admin using this as leverage to bring Apple manufacturing and/or offshore cash back to the US in exchange for covering up this alleged national security issue.

>https://www.businessinsider.com/trump-claims-again-apple-is-...

I'm not convinced which story is strictly the correct one yet but I don't think the strength of the denial is an argument for either. It just shows that it's important that your narrative wins, truth or not.

The burden of proof should be on the side pushing the accusations.

In that sense, Apple’s denial puts it squarely back to Bloomberg to prove it’s not bulshit. Until then it would be fair to discard their narrative.

"before servers are put into production at Apple they are inspected for security vulnerabilities"

A chip smaller than a grain of rice nested between motherboard layers.

What tools could detect such a chip?

While I don't doubt Apple inspects, the zero-day threat always exists.

Xray inspection by a human specialist

Further in the article, Bloomberg mentions a chip, smaller than a pencil tip, sandwiched inside the PCB itself, under other traces. That seems bananas difficult to pull off, but could x-ray inspection find such a thing if it existed?

I specifically mention non-automatic inspection as automatic one may well not have the "paranoid mode," unlike a human who specifically told to go over every individual square millimetre.

Minimum quantity orders for these servers are typically 10,000.

Even at 1,000, I don't see how humans could reliably detect this.

Yes, because the incident they reference was a mid-PCB-layer chip discovered by govt agencies using X-rays. The findings were presented at a private conference in Virginia. Sources who attended said "they've seen the xray pictures" etc.

It is very odd. If there was any truth to the story, I wouldn't expect such strong denial from apple and amazon.

There has been a noticable uptick in anti-chinese news lately, particularly on business oriented news ( cnbc, bloomberg, fox business, et al ). Obviously some elements of the government or power structure has instructed them to do so. But why?

Also, if the IC wanted to put pressure on tech companies, wouldn't they just contact apple and amazon directly. Maybe there is a rift within the IC with one group being pro-china and another being anti-china?

Or is the IC and news companies done targeting social media tech companies and switching their attention to regular tech companies? Just have to wait and see.

Straight up some hardware/embedded nerd that posts on here would have one of these chips in the wild and already have posted a hardware break down blog post under a microscope of one.

They kinda seem like a Tom Clancy fantasy. But I dunno.

Why do you so blindly believe amzn/aapl over bloomberg? You do realize that reputation is a huge part of success, particularly in the security and hardware space. Of course amazon and apple will deny this story. For starters, they are probably under a gag order from the government to not leak anything regarding their findings since the investigation is still ongoing. Secondly, it is of no benefit to admit they failed in oversite on their side with this one.

> Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.

That to me is the icing on the cake. Companies are usually not direct like that.

(Unless this is some kind of new NDA that allows/mandates itself to be denied)

Reply on news.ycombinator.com