How Tier 2 cloud vendors banded together to cope with Spectre and Meltdown | Hacker News Reader

How Tier 2 cloud vendors banded together to cope with Spectre and Meltdown

techcrunch.com

196 pointsedouardb68 comments

The more I know about how Intel has managed the information, the less I trust them. What a disaster. CEO and all the Press and Communication team of Intel should be fired

It's not clear to me what the best approach is here. The wider the circle of those who know, the more likely it is that there will eventually be a leak. Three can keep a secret if two are dead and all that.

Exactly. I assume if one of the smaller providers like Linode found an internal vulnerability that they thought was a big security risk to have widely know and had 3 giant customers and a large number of smaller customers, they’d work directly with the giant customers in advance in the same way.

I was a Linode customer back when there was a security incident where everyone found out on Reddit before the company bothered to tell anyone. And then once news did come out they never said (a) what happened or (b) what steps were taken to prevent it happening again.

I am very reluctant now to trust the little guys with anything remotely mission critical.

I am resisting the urge to make a Linode and ColdFusion joke.

This proverb is mostly not true. I've worked in high-secrecy areas and everybody keeps their mouth shut because they're terrified of the consequences to their career.

Three can keep a secret if two are bound by an NDA is closer.

I think in a tightly bound organization where all incentives are well aligned, that can be true.

There was a leak.

Typing this from Ubuntu - still no updated kernels yet.

It wasn't so much a leak as someone putting two and two together and the cat getting out of the bag.

Anyway, this is a nonsense argument in the context of the thread. They tried to keep things quiet by limiting the size of the circle, among other things. Their attempts at secrecy did not entirely succeed, although they made it 90% of the way to the embargo date without things coming out. Their inability to make it 100% of the way does not inform us at all about whether they should indeed have limited the circle of those who knew.

Should be coming on/before Tuesday: https://insights.ubuntu.com/2018/01/04/ubuntu-updates-for-th...

If I'm not mistaken, this was the same approach taken when dealing with heartbleed.

Can you be more specific? What aspect of how Intel managed this are you referring to?

Why? Intel made billions last year. They will make billions this year. They will make billions next year.

> Why?

Because the fallout will cost billions over the next years. Intel as a company due to class-action lawsuits, recalls, rebates, and the shareholders because the drop in stock value after the announcement will cost them quite a chunk of money.

In addition, more long-term, I sincerely hope that the cloud vendors (and maybe even Apple!) recognize that their total dependence on Intel (and NVIDIA in deep learning...) is bad and they need to diversify their base.

Unclear that diversifying helps solve this sort of problem. More vendors could lead to the same number of bugs, but less investment in quality control per product e.g. if you make $1b and spend 1% on quality control, then you spend $10m checking your product for bugs. If the market fragments into 10 $100m vendors, then to get the same amount of money spent checking each chip for bugs, you'd have to spend 10x as much of your budget on quality control.

It gets even worse when you consider the incentives of private/academic/third party researchers to search for bugs, since there is a lot less prestige in catching a bug in a smaller/less well known product. e.g. I'm pretty sure no white-hat security researcher has checked for security problems in the cheap wifi light switches I bought on Amazon.

> If the market fragments into 10 $100m vendors, then to get the same amount of money spent checking each chip for bugs, you'd have to spend 10x as much of your budget on quality control.

But there's a much smaller attack surface and the incentives for attackers are significantly changed. Homogeneity is always more vulnerable to disaster, whether we're talking about food supply or chips.

And quality is not directly a function of money spent. Customers will start demanding to see the machine checked proofs that your hardware is correct. Intel has been cowboy coding CPUs for way too long.

> Unclear that diversifying helps solve this sort of problem.

At least having the option of another vendor as a fallback (e.g. in case there's a severe RCE vulnerability in ME/PSP) is a better alternative than having to shutter your entire business.

I would not be surprised if these management engines have a backdoor that can be invoked from a guest VM... and then an all-Intel (or all-AMD) shop has a massive problem.

Intel ME drivers are not loaded into a VM, the devices are not exposed and the ME MSRs are also not exposed.

> and the ME MSRs are also not exposed.

This is the core question: is this isolation absolutely perfect, or can it be pierced in any way? Something on a severity level like Spectre/Meltdown - people would have laughed you off the stage half a year ago when you told 'em you could read kernel memory from Javascript without exploiting both the browser and the kernel - is IMHO certain to be present in either of the "management" solution, and I'd like to be prepared when the bomb explodes.

I don’t think any serious security person would have laughed you off for mentioning side channel attacks.

No isolation is perfect but and that is an important but for virtualization you have much higher control over what instruction you allow through so an attach which is specific to ME isn’t likely.

That said you can have a side channel attack that allows you to compromise the hypervisor and from it you can jump to the ME but this is a different story.

> e.g. I'm pretty sure no white-hat security researcher has checked for security problems in the cheap wifi light switches I bought on Amazon.

But isn't this also applicable to the other side? As in, black hats have less incentives to research vulnerabilities in less popular products (security through minority). I'm not certain how this balances out.

Cheap products can be quite popular. It's probably fine until that cheap item you bought goes viral on facebook, or there is a single upstream vendor that is hugely successful. No idea either, only statistics would tell.

AMD chips do not need PTI. All the performance hits people are talking about right now would be irrelevant on a cloud farm that used AMD CPUs for their hosts. If any such cloud farms exist, they had better be declaring that loud and proud; I expect there will be a lot of people looking to jump ship.

Different vendors have different bugs.

The aerospace industry has realized this a long time, and for some things they will have 3 different devices from 3 different vendors doing the same job

The intel CEO took over in 2012 - some of these vulnerabilities go back as far as the Pentium Pro (Meltdown) and the other one, effects (I believe) as far back as the original Pentium (Spectre) - why would you fire someone for something that happened under a predecessors leadership (Andy Grove, in this case was CEO when the Meltdown attack was added) - it makes no sense - I see nothing here that doesnt jive up with similar efforts with other bugs.

Him selling his maximum allowed grant and also options when he knew before everyone is plenty.

According to Matt Levine, Krzanich has a consistent history of selling the maximum or near-maximum amount of his stock grants every year.[0]

If one wants to be truly cynical, you could say he has been expecting something like this all the time and cashing out to ensure his money isn't tied to Intel. I am inclined to being slightly less cynical - I'd say he has been just cashing out regardless of the the company's performance.

0: https://www.bloomberg.com/view/articles/2018-01-05/citi-forg...

That would be clear insider trading, with SEC prosecution and jail time. Unlikely if the executive has half a brain.

Yeah, exactly: it's too easy for people to draw a direct line between the selling and the announcement (which, of course, people have done) for him to have been stupid enough to do it with that intent. He seems to have simply been following the pattern of previous trades.

That being said, even though I don't think it's the case here, greed has been known to make people do some very stupid things.

You do know that INTC is up compared to 1 month ago? If it didn't drop that much on the initial announcement, why would it plummet in the coming months?

Because there is no quantifiable impact yet. Right now all that's known is "mh, it's bad, but the OS vendors are patching it"... now give the situation a couple weeks to brew, wait for more data on the CPU impact of these patches and the inevitable lawsuits. Plus, Intel might want to think about delaying the next CPU releases (or introduce a new stepping of existing CPUs) to fix the bugs in hardware... all stuff that ain't cheap.

If the market agreed with you that Intel will suffer greatly in the future due to Meltdown/Spectre the price would have dropped already. Of course you might be correct and the market wrong. You’re short Intel, right?

> Because the fallout will cost billions over the next years.

Doubtful. Intel has a near monopoly in data centers in 2017. They will have a near monopoly in data centers in 2018 and 2019 I predict as well.

Really very few things they can do to lose their business at this point.

Until Intel fixes their architecture, wouldn't all the major cloud providers switch new purchases to AMD for the time being?

The problem is chip R&D is super capital-intensive and the space is highly competitive even with so few companies.

Shouldn't the CEO be investigated for insider trading?

The CEO should probably spend some time in the pokey for all the stock he sold between when he knew about this problem, and when it was publicly known. I'm curious how many other Intel employees sold stock in the same period.

This has been debunked already - CEO sold the maximum amount of company stock he could from his yearly award every 4th quarter for the last 4 (5?) years in a row.

Incorrect actually. Though it is true that Krzanich has consistently sold stock every 4th quarter, for this recent transaction, he massively increased the amount he sold. Krzanich sold so much in fact that if he had sold any more stock, he according to Intel corporate mandate bylaws would no longer be allowed to continue as CEO.

If you don't believe me, look for yourself: https://www.fool.com/investing/2017/12/19/intels-ceo-just-so...

https://www.sec.gov/Archives/edgar/data/50863/00011276021703...

And 'coincidentally', this massive increase in selling just seemed to somehow perfectly fit right in the middle of Intel being privately informed of Meltdown in June but before any public disclosure kept locked away all the way until now in January.

This is blatantly illegal insider trading. Don't let anyone tell you otherwise.

It's not clear to me that he's behaving like he did in the past:

> According to filings, on Nov. 29, Krzanich exercised and sold 644,135 options and sold an additional 245,743 shares that he already owned.

-- https://www.bloomberg.com/news/articles/2018-01-04/intel-ceo...

Sure, he sold most of his 279k grant, just like he sold most of his previous (much smaller) grants, but he also flipped a huge pile of options. This strikes me as the behavior of a man who never had much confidence in the company he ran, and suddenly had much less.

Mottley Fool article by Ashraf Eassa on December 19 [0] supports your position that it wasn't just routine behavior:

>However, there were two transactions that Krzanich reported in that Form 4 filing that I thought were more notable than typical stock option exercises and subsequent share sales. Let's take a closer look. ...

(Assuming that Ashraf called this without inside knowledge of what Intel had already disclosed by that date to selected 3rd parties.)

[0] https://www.fool.com/investing/2017/12/19/intels-ceo-just-so...

I would do the same thing no matter the stock. It is stupid to keep all your eggs in one basket (as others are saying about userland all through this thread).

He is not inspiring much confidence in the future of INTC.

This does not protect said CEO from breaking the insider trading laws. Nor does it protect Intel as a company from conveniently "omitting" the elephant in the room in their press releases and financial risk statements.

Rules against insider trading do not prevent executives from trading the company's stock. They can set rules with their financial advisors like "sell x amount of stock every y weeks" which will be executed regardless of insider knowledge the executive has.

Saved you a click: by starting a shared Slack for their teams to collaborate.

Neat factoid but this article is not exactly information-dense.

I don't think the actual technical "how" is of much importance so the article doesn't spend much time on it. They communicated effectively, and more important openly, over company boundaries. Many tools would have worked for that.

This is about "how" they banded together to reduce their disadvantage compared to the big cloud providers that had advance warning and insider knowledge/access to the vendors long before the rumors started.

Correct. It's the ability to share documents and conversation snippets provided by vendors, as well as curating and summarizing the significant amount of information available.

I've never seen an exploit that involves microcode updates, compiler fixes, kernel patches, and KVM/Xen updates all together. The number of moving parts is staggering.

Being able to filter and summarize that across company boundaries has helped me both understand and more effectively work to mitigate this problem.

And now there are "Tier 3" companies that didn't get invited to the slack channel.

It all helps big get bigger and making life harder for smaller companies. Not a great setup for a healthy competition.

Prgmr.com has been participating on the Slack channel this article mentions. Being able to share notes gleaned from reaching out to vendors and sort through the information and mitigations for Spectre and Meltdown has been a huge help.

I work for a cloud computing company that would have benefitted (and hopefully contributed) from being on that Slack. If you have the time, would you mind reaching out to me with details? My email is in my profile.

Here is where I asked to join:

https://twitter.com/prgmrcom/status/949023633581592576

Following up to @scaleway on Twitter in the same manner would hopefully work out.

I've sent you an email with a bit of elaboration on this.

Do you think the communication channel will persist beyond the current vulnerability? E.g. give it a name, allow other cloud providers to join?

actually this kind of channel existed before (with OVH at first for the last few months, to fight some big abusers/phishings together) I guess it will depends on everyone's good will, but most of us will probably stay connected, it does not cost much to keep a slack tab opened (well a few Gb of memory haha, thanks Slack) ;)

What's the advantage of Prgmr?

You can manage your VPS over SSH using our management console.[1] Since most of our users access their VPS using SSH, it's nice to use the same protocol for powering a machine off and on or accessing the serial console.

By and large our customers know what they're doing. That lets us provide IRC and email-based support that has been described more as working with a colleague than interacting with a vendor. This can be helpful when, for example, a user self-hosting email receives a complaint or has a delivery issue.

Most of the time though we just get out of the way and let you work on your VPS.

1: https://wiki.prgmr.com/mediawiki/index.php/Management_Consol...

I’ve said many times this is the best way to deal with the large monopolistic companies - group up.

This is a great example of collaboration between competitors where there is much to be gained and nothing to be lost!

All this seems like a pretty compelling reason to move to using VMs and type safety to provide process isolation instead of using the hardware, e.g. https://www.destroyallsoftware.com/talks/the-birth-and-death... or Microsoft's Singularity and Midori projects.

It's a shame there's so much inertia behind the current setup of hardware memory management etc that it seems it'll be a long time before anything actually happens here.

How would this approach have helped in this case?

As I understand it, the flaw in question allows reading kernel memory by executing user space code. How can a layer of software, on top of this type of buggy hardware, fix this issue?

In one sense, the issue has already been fixed by a layer of software on top — which restricts a bunch of stuff, and reduces performance — but I assume this isn’t what you’re looking for.

Those systems don't use the buggy aspect of the hardware (memory protection) at all. Instead, all code is run inside a VM which provides memory protection and process isolation - there is no 'native' code at all.

Not using the hardware memory protection provides a ~20% performance boost, which makes up for the ~20% overhead of running everything through VMs.

FYI, the submitter is someone from Scaleway, so he perhaps would also answer some questions on what infos they had or something.

everything we have at Scaleway is on our blog http://blog.online.net things are calming down now, we known what are the fixes for the known bugs and their limitations We can probably expect more OS patches in the coming weeks to improve performances and potentially fix new variants of these bugs that could come in the future

Then suddenly he said, “I have it! We are going to swim all together like 0.1% of the biggest fish in the sea!”

~ Leo Lionni, Swimmy

Note that Google released information a week early, giving attackers a leg up in attacking these cloud providers, as opposed to Google giving the cloud providers the information immediately, which would have helped them defend themselves. But, surprise: Google is their competitor.

Google is basically the new Microsoft, except Microsoft could actually design working products when it was evil.

What's with these random jabs at companies, you are just trying to create a culture of irrational dislike. I don't even care if you attack Google but everytime you do so, you have to cite exactly why you are doing it. Not some nebulous "oh they cant even make a single working product" which is objectively false and just provocation.

One, you're attributing something I never said, and two, my actual claim is not objectively false. Three, I don't have to do anything, much less prove what is otherwise easy to find out via Googling. And four, it's not irrational dislike, it's dislike based on personal experience and observation. Five, it should be pretty obvious exactly why i'm attacking Google, it doesn't need citing (unless the reader can't put two and two together, vis a vie "Google uses its privileged embargo status to disseminate sensitive information in a way that harms its competitors")

Reply on news.ycombinator.com