Hacker News Reader

Comment by hendzen | Hacker News Reader

Apple and Amazon probably have a small set of TS/SCI cleared employees who dealt with this mess. It’s likely 99.99% of the employees at those firms had no idea what was going on. The switching out of thousands of compromised servers was probably made to look like routine maintenance or upgrades and the whole affair was kept secret. That is, until some high level government employees intentionally leaked it to the media, probably under direction of the White House to garner support for a more aggressive stance on China - the trade war in particular. Read between the lines.

View on HN

That's a lot of unsubstantiated assumptions. It's also explicitly denied by both Amazon:

>It’s untrue that AWS knew about a supply chain compromise

and Apple:

>we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them.

There's no in "between the lines" or ambiguous wording there, they flat out deny it. Unless this small set of "TS/SCI cleared employees" worked completely on their own without reporting to anybody else in the company this means that they are lying in these statements.

It's possible that they do just that but it's a bit strange to me that they wouldn't find an easier way to deflect the issue without using such a strong and explicit language. Something vague like "we've been working with the authorities and have no reason to believe that any sensitive information has been leaked etc..." would be easier to spin if it turns out that somebody can prove that these attacks took place.

Surely if the scale of the attack was as large as reported by Bloomberg in their article it should be possible to find one of these backdoored boards in the wild? Or at least have testimonies by employees in these company that could testify that batches of motherboards were suddenly replaced for no obvious reasons?

And if trade war is the reason why deny it now? What do they have to gain from that, they're the victims in this story as far as I can tell.

Those employees wouldn't be authorized to talk about the information to anyone not cleared. Which includes the PR department or anyone tasked with crafting a statement.

Any TS/SCI cleared employees must deny the existence, even to their manager or even the CEO, unless they are also TS/SCI cleared.

It'd be interesting to know how that interweaves with things like SEC requirements to disclose material information.

Generally (and unfortunately) "national security" tends to override everything else.

Where did you get that from? This is well known to not be true in the US, the government can't even stop you from publishing nuclear secrets or the Pentagon Papers. See New York Times v. United States (1971) and United States of America v. Progressive, Inc.

That's not how TS/SCI works. You've been watching too many movies. Source: was TS/SCI cleared for 15 years. Information being classified just means you can't divulge the information, not that you can't divulge the existence of the info.

That's a bit misleading. There are code word programs and classified investigations where just revealing the existence of the info would be a serious violation.

In this case, if a cleared employee is asked: is this information true, is there such an investigation? Then simply by saying they can't comment on the question, they reveal the info to be true.

I think most people with high-level clearances would play it safe in such situations and just deny any knowledge of the situation.

I can't comment is exactly what they can and do say very frequently. Apple could have not commented on any investigation or not responded to a request for comment. This would have been completely normal. "That's not something I can talk about" is a very common phrase.

Frequently is not the same as always. And, if true, this situation would almost certainly fall outside the common scenario.

Absolutely correct, but extraordinary claims require proof. I'm certainly willing to be proven wrong, and of course I could be, but there is no indication that this is anything other than Bloomberg being misled at this point. Location of clearly exploited hardware, acknowledgement by anyone involved, or analysis of traffic from one of these boards would all corroborate the story.

I would say it depends on the thing.

I worked at Defense Science & Technology for a year, and there were levels of classification I wasn't even cleared to know about.

It was very common for someone to be working on something they could not tell their direct boss about, because their boss was not cleared for it.

>>just means you can't divulge the information, not that you can't divulge the existence of the info.

The existence of it is important. I can't tell the content of x letter sent from China's Amb to the mother ship but I can tell you that we intercepted the letter. How was it intercepted? That in itself means a lot.

>> >It’s untrue that AWS knew about a supply chain compromise

Funny they didn't deny that it happened, just said AWS didn't know about it.

Exactly, this type of weasle word shit is littered throughout all of the statements.

Same paragraph, they state:

> We’ve found no evidence to support claims of malicious chips or hardware modifications.

That's about as good a denial as you'll get from somebody with a lawyer.

“We’ve found no evidence” ⇒ Somebody else found the evidence.

This is fun.

You can read any sentence to mean anything else if you want, but all you'll get is a degree in literature.

“AWS didn't know about it” ⇒ The relevant part of Amazon is not in the AWS organization.

I mean... yeah of course. There's always a chance that something happens and you don't know about it.

I found this wording interesting:

> We did not uncover any unusual vulnerabilities in the servers we purchased from Super Micro when we updated the firmware and software according to our standard procedures.

Does that mean they did uncover some "usual" vulnerabilities?

Typically yes. This is base hardening. The usual vulnerabilities can be just default configuration options or unwanted firmware/software features.

Similarly no security team is going to say "There are no vulnerabilities in the servers we purchased." It's just not true, they're always there and expected.

There's enough "usual" vulnerabilities that the practice of fixing them has a name: hardening. For supermicro motherboards specifically, "open services with really shitty default credentials" is apparently one of them — definitely one that is common in a lot of other contexts.

In this context, "unusual" vulnerabilities would be evidence of a deliberate attack, rather than just common security mistakes.

Most of the out-of-band management systems (aka IPMI/DRAC/LOM) that server vendors use are built by the same 2-3 companies and rebranded.

These companies churn out some truly horrible software with little consideration for security.

It's often difficult to automate firmware updates, so they tend to stay vulnerable.

It's a similar situation to webcams: https://youtu.be/B8DjTcANBx0

or they uncovered it using non-standard procedures?

> Or at least have testimonies by employees in these company

The original article directly addressed this: "The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."

It is entirely likely that the companies affected were directed by the IC agencies working on this not to discuss or reveal their knowledge of the hack. Often in intelligence operations it is important and useful to not alert your adversary that you are aware of their intrusions until you are fully ready to take action against them, or have fully removed the danger.

I don't see any reason to take the companies' categorical denials as evidence that this did not happen or that they were not targeted. Those statements are what one would expect in a national security incident and investigation of this magnitude, with such serious implications.

> this means that they are lying in these statements.

Cyberwar with China is the current equivalent of the old Cold War scuffles with the USSR in third countries. How many lies were told back then, to cover up the worst mishaps? And then they were "uncovered" when it was safer or more convenient to do so.

I can totally believe that there are tons of actual conspiracies regarding the tumultuous relationships between China and the USA but in what configuration does this particular one make sense? As far as I can tell the only people who come out badly out of this are the Chinese intelligence agencies and the complicit Chinese manufacturers. Why would the Apple and Amazon go out of their way to protect them? I can imagine that the US government could coerce them into doing that but to what end?

If these companies are caught red handed lying on behalf of the Chinese or US governments it would set an absolutely terrible precedent, I don't see why they would risk it when they have so little to gain.

> As far as I can tell the only people who come out badly out of this are the Chinese intelligence agencies

The richest and most powerful American companies, some of whom are critical to national security and/or make their operational security as a critical selling point (the CIA among others), would be found not to be in fundamental control of the essential infrastructure powering their core business. It would trigger expensive large-scale review of every server in the world, and investors would run for the hills. This would be massive, and have heavy repercussions in the markets. FAANG are not protecting the Chinese, they are protecting their own finances - and the authorities will let them get away with it because the alternative is unpalatable.

If you keep it as a denied rumour, there is an official excuse for people to just get on with business as usual - nobody wants to deal with a market crash, not even most traders, and after all newspapers say many things, not all of them true. Maybe Supermicro is compromised, and maybe if you really care about hardware security you should buy elsewhere <wink-wink>; but it's not official, so most people can just pretend nothing is happening and go about their day, until a solution can be found.

> I don't see why they would risk it when they have so little to gain.

They have everything to lose, though.

Just wanted to thank you for posting this, as I [self-censor].

I'd like try to boil things down a little further for people:

1) Most things in the modern world require high degrees of trust. Once a significant portion of people begin to question the system, it fails.

2) The main goal of most organizations/governments, in general sense, who find themselves in a favorable position is to keep the game going.

3) Though it's impossible to control for all variables, 'we' believe we can manage most common ones. The uncommon (foreseen and unforeseen) often arise in times of crisis (panic behavior) and often are unwieldy.

So... the veracity of the story is generally less important than is managing reactions to it.

4) IANAL, but I worked w/them for many years and have crafted many statements. Communication is an art form open to interpretation.

5) I've long suspected that such "tampering" was standard practice, for any global power. Why? See point 2.

I encourage anyone skeptical about any portion of the story, to don his/her 'megalomania'-cap for a bit. Then everything, at least conceptually, should make sense.

Cheers!

Great, this means humans are going to Mars now.

The language is not very explicit. They allow the reader to interpret thier statement through an implied explicit denial. Great liars if not just plainly unclear.

This whole thing wreaks. Is it fake news? Are the statements from Amazon and Apple expertly crafted to fool people who aren't very good at reading comprehension (most people)?

See, this is how FUD works. It works.

An "implied explicit denial"? How does that work? Can you point out how these statements leave open the possibility that they know about this attack?

There is a lot of information in the Bloomberg story, which I mean in the information-theoretic sense, and not in the sense that it must be "true" information for any definition of "true". A lot of very specific claims. Those claims did not come from the reporter writing the story, which is about 99.9999% likely to be incapable of making up such a plausible story with such details.

So one must think about "Where did this information come from?"

It is at least a plausible theory that the story is largely true (though as I posted in another thread, I'd bet money it's not all entirely true), and that the denials are either made by people who are unaware of the truth, or are being made deliberately. (Actually, the people who literally prepared those statements almost certainly believe the truth of the statements. One of the most plausible ways for a group of people to lie is for all communication to come from individuals who genuinely believe the lies; no body language tells or any other such leakage about it not being true.)

It is also plausible that the denials are in fact true, in which case one is left with the very interesting question of "Where did all that information come from and why is it wrong?"

As a couple of people have also said, there's also the option that the story is largely true, and the denials are true if you parse and read them like a lawyer, but meant to mislead anybody who doesn't. I can't say I've examined them for that, but it's definitely a possibility to consider.

I'd actually suggest "propaganda" isn't a great explanation; propaganda does not generally depend on making lots of specific, refutable claims, and certainly not followed up by refutations immediately. It is usually designed to speak directly to people's emotions and fan pre-existing flames in ways specifically designed to not be refutable. If this really was government propaganda (note how that is more specific than my previous unqualified "propaganda", because anybody can propagandize, not just governments), I would expect the American companies would be strong-armed into agreeing with the story for the propaganda's purposes, or that the story would never run at all if they couldn't be sure the companies weren't going to back them.

An alternate theory that might fit all the facts is industrial espionage. Let's say the story is completely untrue, the denials completely true. What is the result of this story? Supermicro in particular stands to lose some business. Perhaps someone who benefits from that planted this story.

Another alternative is stock market shenanigans. As I write this, the delayed feeds aren't showing it yet, but Supermicro stock (SMCI) just took a 31% bath overnight. Who benefits? Short sellers, put option traders.

I have no idea myself. "Unknown unknown" is still a pretty large chunk of my personal probability estimates.

Supermicro was already delisted prior to this article and had about $370k of US dollar trading volume yesterday in the OTC markets. Don't think people will make much money shorting it as this point, since it is already so illiquid. Highly doubt that is the explanation for the article.

EDIT: I read the volume amount incorrectly - it's more like 6m of volume which is not too bad. Someone could probably make money shorting it but again since its delisted already it would be difficult/risky to do so.

Thank you. I was trying to make sense of the news stories listed below the stock listing.

Whatever is going on with whatever stock is being traded is now visible even in the delayed listings; you can see it on https://finance.yahoo.com/quote/SMCI?p=SMCI&.tsrc=fin-srch or your choice of provider.

I wonder if their delisting is related to this hack, or if it's a sign that they have poor internal controls and the company is just poorly controlled every where from accounting to engineering.

> Apple and Amazon probably have a small set of TS/SCI cleared employees who dealt with this mess.

For those who are wondering:

TS: Top Secret: https://en.wikipedia.org/w/index.php?title=Classified_inform...

SCI: Sensitive Compartmented Information: https://en.wikipedia.org/w/index.php?title=Sensitive_Compart...

Security employees don't contact the FBI without consulting or at the very least informing the higher apps.

PR departments don't vehemently deny, on point, an article from a major newspaper without consulting the higher apps.

There is literally no chance that this happened and the response was made in ignorance.

Either Apple and Amazon are lying or Bloomberg is wrong.

> Either Apple and Amazon are lying or Bloomberg is wrong

Apple and Amazon are compelled to lie. It's a classified investigation and likely only cleared employees are aware of it, and they have to deny its existence or stand to lose their clearance (worth $$$).

How can companies be forced to issue such press statements? They may be forced to give incomplete answers when asked, but these statements go far beyond answering questions.

The people issuing press statements are almost certainly out of the loop. And of course, no one with knowledge of the situation is going to step forward, given they have a strong incentive to stay silent.

I mean, think of your own company's PR team. If you've worked for a large company, you've likely had interaction with a PR team that was pretty ignorant about the inner workings of the company. Now, add to that the fact that this is a classified investigation being run by FBI counterintelligence, and interfacing only with cleared employees (none of whom work for the PR team).

You are a PR guy. A respected and big news agency puts out an article saying your company is involved in some secret stuff - stuff that you wouldn't know about.

Do you release a press release denying everything without asking the guys who are supposed to know about it? Would those guys say to you it's all fabricated and urge you to deny it or will they evade answering it (or even more likely they'll go up the chain)?

This is a CEO level incident.

> This is a CEO level incident.

And as the article describes, the CEOs were summoned to DC, briefed, and at that point almost certainly asked to keep the investigation under wraps.

Issuing such press statements is the opposite of "keeping the investigation under wraps".

Right, PR folks may not understand how all the stuff works in their company, but they sure as shit know which topics require senior approvals. That is basic self-preservation for them.

When did the US turn into a dictatorial state? I thought the US was free country.

Their denials are very, very specific. Meaning those words can be true, and a blanket denial would be false.

Please note that the US government cannot compel speech and - by extension - cannot compel anyone to lie.

Of course it can. The methods it can do so legally are limited, and it has no "free floating" power to do so, but that doesn't mean much.

Provides only minimal substance here, but I'll mention I have worked in a company where there were times I knew information I wasn't even allowed to share with the CEO of the company. He knew who our customer was, but he was not allowed to know the substance of the work we were doing due to the non-disclosure agreements.

What I'm trying to get at is, it is possible for an agreement to be created for someone internally to know some "important" information in a company, and for it to not be known by others, even those higher than the employee.

100%. Work at a company that provides equipment to the government. There where people that have a clearance and know why and how thing where being used, but the exec staff up to the CEO just knew who the customer was and no details beyond. It’s prtty normal.

So do you have anything to support these fairly fantastical claims?

In corporations you don't address the federal government without your legal team in the conversation, and your legal team would also be in the approval path for any public statement about such an allegation. It's extremely unlikely that a public company would comment on something like this without connecting the dots internally first.

Plus, they gain nothing by admitting that China hacked them. Your data is there and all.

I got a personal text from Trump yesterday confirming this.

I simply can't decipher this comment

Yesterday in America, an SMS message went out nationwide from FEMA. It was labeled a 'presidential alert' so many people assumed it was Trump that sent it. This went viral on social media and people began photoshooting funny messages that Trump or other theoretical leaders might send.

Normally I'd assume this sort of leak would only be the result of orders from above. But there was recently the credible allegation that Trump backed off the ZTE sanctions after China threw a few million dollars at the Trump Organization in a real estate deal[1]. Whether that was a successful case of bribery or not I can see a highly placed employee looking at that and deciding they can't trust their superiors to do the right thing in this matter.

[1]https://www.businessinsider.com/trump-zte-order-after-china-...

Reply on news.ycombinator.com