Meltdown and Spectre Linux Kernel Status
kroah.com
kroah.com
Red Hat built their own mitigation for Meltdown and Spectre and were the first distribution to have an updated kernel right when the embargo was lifted.
As far as I can tell, their meltdown mitigations are similar to what's in the upstream kernel now, and the Spectre mitigations relies on Intel's new MSRs (the microcode update that had since been retracted - new microcode_ctl packages remove it).
Interestingly, you can enable/disable the mitigations at runtime with Red Hat, which is useful for performance testing. I found no way to do this with the upstream kernel.
I'm not happy that Intel, Google, Red Hat and a few others have known about the bug for months and the upstream kernel devs are scrambling to build mitigations and still don't support the microcode mitigations. Google apparently patched their stuff in September, while everyone else is left out in the cold. This is a real mess. Surely there must be a better way than telling a few select companies?
The Red Hat guys in particular surely would've loved to collaborate with the upstream kernel. Must be frustrating not to be able to tell anyone.
the worst case would have been someone in the latter set leaking the info, which is just what we have now. So it could have been potentially better. There was really no reason to drop a bomb like this.
Issues on a similar scale have been properly handled in the past: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-...
> The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning.
> Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered.
I think Colin's take[0] was interesting and insightful there, especially in the light of him having handled a very similar issue a decade earlier:
> The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise.
> Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered.
> […]
> To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs.
(emphasis his)
[0] http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-... final section on vulnerability disclosures
Linux is a loosely organized community of developers who collaborate in public. They don't have any ubiquitous communication channels established other than mailing lists. If you attempted to establish a private security mailing list, the list of people who get to go on it is (1) hard to define (2) very large and (3) frequently changing. The affected subsystems have lots of disparate maintainers and interested parties who need to get their eyes on any patches that go into them.
It might not be ideal from a security person's perspective but from everyone else's perspective this model has been the driving factor behind the most successful collaborative project of all time.
Bullshit. Ultimately, Linus decides what does or does not go in, assisted by the subsystem maintainers he picked. If they wanted to have closed doors meeting, they could, and would.
FreeBSD and the like have no more and no less "single organisation calling the shots", the only difference is that they're not a human clusterfuck.
> It might not be ideal from a security person's perspective but from everyone else's perspective this model has been the driving factor behind the most successful collaborative project of all time.
That just means Linux at large will ultimately get ignored for anything security-sensitive, and only specific distributions will be contacted.
> The FreeBSD Project is run by around 500 committers, or developers who have commit access to the master source code repositories and can develop, debug or enhance any part of the system.
I doubt there are 500 people with push access to the main Linux repository. Again, the FreeBSD folks just have their shit together.
Of course there is. Linus decides what goes into the mainline kernel, his repository is literally called "Linux kernel source tree"[0], with the stable "archive" being sensibly called "Linux kernel stable tree"[1], and Linux's analogue to the freebsd core team is de-facto Linus and his trusted maintainers, the people from whom he reviews and pulls commits.
[0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
https://github.com/torvalds/linux/blob/master/MAINTAINERS
Linus might have the "final say" but all of the work and knowledge is encapsulated in these people and they are generally given the autonomy to organize and maintain their subsystems as they see fit. As for Linus's tree, it's probably the most Linux Linux that's out there, but it's hardly the only one. Hardly anyone actually runs Linux from Linus's tree. Most distros maintain their own trees with their own patchsets.
I perfectly understand that you're trying to excuse the utter clusterfuck that is the linux project's handling of security notifications, apparently on the grounds that they don't have their shit together and refuse to do so, ever.
> Most distros maintain their own trees with their own patchsets.
Yeah, and they can actually be trusted to not break embargoes.
For details on how this works, please see the kernel documentation itself: https://www.kernel.org/doc/html/latest/admin-guide/security-...
>It usually does this by just dragging in the responsible parties, but many times it just fixes the problems themselves and submits the patches to the proper part of the kernel.
I think this is the key - once the patches are submitted they go through the normal workflow and aren't any different from a typical patch, right?
Personally I don't think this is a huge deal, but it's where the disconnect between the security person's ideal worldview and the reality of how Linux is built colllide.
Like the linux-distros [0] list? Or just the kernel developers?
(Note that both Linus and Greg are on this list.)
[0]: http://oss-security.openwall.org/wiki/mailing-lists/distros
If FreeBSD was contacted early on, would we be hearing these gripes from some NetBSD or Dragonfly BSD developer? Maybe that's actually a reason BSD was contracted late, maybe they expected interested scrutiny from the other BSD projects from any kernel patches. Linux certainly ended up exhibiting that behavior, maybe they were worried about it there too but Linux's size meant a different call was made?
I understand why BSD devs are upset, but the complaining comes across as exactly what it is, whining they didn't get treated like the big boys. It sucks. Linux used to be right there with you, and neither got any special treatment, but times have changed.
I'm not at all anti Google saying "We are Google. If you pay us to run your systems you would be protected. If you do not, then eff you, we don't care". I'm anti-fluffy "Google is awesome and does things for the community" garbage popular in tech circles, HN included.
All you're really doing here is confirming that you are typical of the HN userbase these days (far past its prime and filled with reactionary tinfoil hat types with a raging hate-on against one company or another) - that Google is on your personal shit list, and that you'll characterize just about anything positive about them as "fluffy garbage". And that's just not very interesting - it's tiresome, practically-writes-itself noise to everyone other than the choir you are preaching to.
I am prefectly OK with Google and Microsoft's position being "Eff you, we only support our customers and those who are not our customers can become our customers so we can support them".
The general view on HN for some reason is that Google this benevolent do gooder. That's the view that needs to be dispatched. Google is a standard company. It happened to be quite big and have quite a big market. To defend its market it needs to be involved in security research. For lots of reasons, of which I would say PR is one of the most important, it also cannot afford to sit on the vulnerabilities that it discovers forever as someone else may discover them independently ( think Chinese competitors of Google ) and may use them against Google's interests in a way it cannot currently identify.
HN user base gets lots of Googler who are pretty fanatical about the company -- which I guess is understandable as without Google they may not be able to support their lifestyle.
Not only is that a hypothetical what-if scenario that you've invented whole-cloth to support your argument, it's provably false since there was parallel discovery from three other teams at approximately the same time.
https://www.wired.com/story/meltdown-spectre-bug-collision-i...
FreeBSD is a serious project which takes security embargoes seriously, they don't break embargoes: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-...
> I understand why BSD devs are upset, but the complaining comes across as exactly what it is, whining they didn't get treated like the big boys. It sucks. Linux used to be right there with you, and neither got any special treatment, but times have changed.
Not notifying the FreeBSD devs is bad for the entire internet. The Netflix open connect CDN is FreeBSD and pushes 37% of North American downstream traffic[1]. Limelight one of the big CDN providers use FreeBSD for it's edge network[2]. Some of the DNS servers for .com and .net are FreeBSD[3] Juniper routers run FreeBSD[4] PS4s run a modified FreeBSD[5], the Apache foundation runs a lot of there infrastructure on FreeBSD[6]. Whatsapp with it's billion plus users run on FreeBSD servers[7].
None of those things should be vulnerable to known bugs if we want a safer internet. So notifying the FreeBSD devs 6 months late did the entire internet a disservice regardless if you use FreeBSD directly or not.
[1] http://appleinsider.com/articles/16/01/20/netflix-boasts-37-...
[2] https://www.freebsdfoundation.org/testimonial/limelight-netw...
[3] https://www.freebsdfoundation.org/testimonial/verisign/
[4] https://www.juniper.net/documentation/en_US/junos/topics/con...
[5] https://www.theregister.co.uk/2013/11/16/sony_playstation_4_...
[6] https://www.freebsdfoundation.org/testimonial/the-apache-sof...
When it comes to operating systems that have a large base of users and may run unvetted code, the big candidates for that are operating systems that have home users downloading and running software or in the browser, and shared environments such as shell servers, shared hosting providers and virtual services providers. FreeBSD's common industry use cases and low end-user use mean it's much less exposed to this particular problem.
Edit: grammar, typoo and clarity fixes.
If FreeBSD should be notified in this case, even though they aren't really in most of the high impact areas, should OpenBSD be notified? What about QNX? What about OpenSolaris? Apparently Joyent got late notification as well[1], and since they actually are a cloud provider, I think they have a much better case for being notified before FreeBSD.
In the end, if there's an information embargo there needs to be a decision about who's in and who's out, and most times there will be some excluded party that really wishes they were included. You can't make everyone happy, but I don't think FreeBSD really needed a huge advance warning, given what it looks like Google's criteria was for notification. The fact the hyperthreading bug that keeps being referenced was handled differently seems to me to be mostly an artifact of a BSD developer finding the bug.
1: https://help.joyent.com/hc/en-us/articles/115015938847-Secur...
FreeBSD has conformed to embargoes in the past, telling them would not harm the embargo.
> but I don't think FreeBSD really needed a huge advance warning, given what it looks like Google's criteria was for notification.
You have source for their criteria?
So? Should they be notified of Microsoft windows vulnerabilities just because they've shown they can handle themselves? There has to be a cutoff somewhere. I don't think FreeBSD was in the highly affected subset of use cases. I can imagine others coming to the same conclusion.
> You have source for their criteria?
No, that's what "looks like" in my comment is there for, to imply it's an assumption on my part. They notified chip makers, the large end user OS makers, and many/most the major cloud providers (this probably depends on what you consider major, I don't know if Linode or Digital Ocean were notified, but AWS was, and Joyent wasn't..), which are what I would think the highest impact areas are, as those use cases either require running third party code or often due so (end users and downloaded software or browsers with JS).
To me, that appears like they tried to get the most major players most affected some info.
Seriously what the hell are you talking about? I was saying FreeBSD should be notified of security bugs that effect FreeBSD and not 6 months after other Operating systems are told. I said nothing of Microsoft.
Of those cases where it is more susceptible, what percentage of the overall likely susceptible hosts does that account for? Do you think it's even 0.1%? Even 0.01%?
If that's the case, and I was Google, I'd be just as likely to notify FreeBSD about this when attempting an embargo as some random MS vulnerability.
The same cannot be said for netbsd or dragonfly. To call FreeBSD a "hobbyist OS" shows a distinct ignorance of the market.
I mean, should OpenBSD have received advance notice? What about QNX? What about all those embedded POS system providers? FreeBSD accounts for a very small percentage of high risk systems, and I think that explains this.
Off the top of my head can not think of any big vulnerabilities found in the last couple years not discovered by Google.
They pay engineers to find these and also pay bounties. This helps everyone including Google in the industry so would definitely still call them a nice guy. Maybe even a really nice guy?
It is very difficult with vulnerabilities what you share and when. Once Spectre and Meltdown was shared we have many PoC also out. So script kiddies all set up to exploit. So it is not simple but have zero idea how you could suggest some how Google is a bad guy here?
Also Google has also shared mitigations for Spectre that looks like a better approach to the broader community versus keeping as a competitive advantage.
They even shared with Amazon a company that has banned ALL companies on their marketplace from being allowed to sell some of Googles products. Personally I am not nearly that nice.
Shellshock?
Original discovery was by Florian Weimer. Many of the Shellshock family bugs were discovered[ß] (IIRC) by Michal Zalewski, after David Wheeler announced that bash was likely infested with a whole bunch of additional parser bugs.[0] MZ is Employed by Google.
Now, I think I recall MZ sharing at least some of the details of shellshock family bugs on his blog, so whether it goes under Google's umbrella or not might be subject to interpretation.
I believe it's safe to say that the original bug was not discovered by Google, but it's subsequent manifestations, after the first round of fixes, were.
0: https://en.wikipedia.org/wiki/Shellshock_(software_bug)
ß: can you say you "discovered" a bug if you are hunting on an already ripe ground with a fuzzer?
(a) That Google didn't notify more operating systems about the vulnerability early.
(b) That one OS community who Google did notify early broke the embargo, leading to exploits circulating prior to the embargo date.
But these are two sides of the same coin. The more groups of people you tell about the vulnerability early, the greater the risk that someone is going to break the embargo.
For a bug like this, it seems like no matter how you go about disclosing things, somebody is going to criticize you for it. Either you didn't tell enough people and the ones you didn't tell are mad. Or you told too many people, and someone breaks the embargo and everyone is mad. Or in this case, both!
No, they are not.
> The more groups of people you tell about the vulnerability early, the greater the risk that someone is going to break the embargo.
Groups have different levels of trustability, disclosing to more trustable groups is less risky than disclosing to a single untrustable group.
So no, there's no better way. You tell every organization that's capable of patching their kernels privately and keeping the vulnerability under wraps - something that mainline Linux can't or won't do, for better or worse. This is like complaining that you found something in the last place you looked - the linux mainline were the last people who heard about it before it went public because as soon as the linux mainline heard about it, it became public.
Everyone here acted responsibly except AMD, who broke embargo so they could force a response from Intel at CES and sell more processors.
I haven't checked, but I'd be surprised. Basically all the backports except 4.14's are quite different from upstream's.
Download the latest archive from Intel at https://downloadcenter.intel.com/download/27431/Linux-Proces... (currently microcode-20180108.tgz) then:
tar -xzf microcode-20180108.tgz
sudo apt install iucode-tool # or whatever your package manager is
iucode-tool -tb -lS ./intel-ucode/*
001: sig 0x00010676, pf mask 0x01, 2010-09-29, rev 0x060f, size 4096
002: sig 0x00010676, pf mask 0x04, 2010-09-29, rev 0x060f, size 4096
003: sig 0x00010676, pf mask 0x10, 2010-09-29, rev 0x060f, size 4096
004: sig 0x00010676, pf mask 0x40, 2010-09-29, rev 0x060f, size 4096
005: sig 0x00010676, pf mask 0x80, 2010-09-29, rev 0x060f, size 4096
First the CPU loads an on die ROM of the microcode it was released with on it's own. Second the BIOS/UEFI may load an update into the CPU before boot if it has a newer one. Third linux and some other OS will load any newer microcode into the CPU at the bootloader stage before the kernel loads.
This is also commonly the case for other modern hardware, not just CPUs... although I expect it doesn't need to happen before the kernel loads and is probably bundled into their respective drivers.
[1] - http://www.zdnet.com/article/meltdown-spectre-intel-says-new...
Although you can also install microcode with the iucode-tool, the example above only checks the downloaded archive.
I wonder if the "cabal" you mentioned actually broke those laws?
1. make sure all tier 1 companies involved are equally patched and on the same page as far as mitigations go before EOY, then involve tier 2 companies (see news on "the impromptu war room") a full month before the embargo deadline
2. not commit or patch anything or fly commented diffs around in the freaking public LKML; git branches to be tested and secondary mailing lists can be private if only for a brief period of time
That's definitely not what happened, and probably the dirty details have not surfaced yet. I can't wait for this to be reasonably over so we can start to read post mortem on the whole process.
When the longer time for fixes is done, post an announcement in private pre-disclosure lists.
For a very complicated change, give a few days of private testing and provide patches and details to groups that must apply the patches (oss-sec distro list, etc.)
For a simple one with existing backports, notify that <DATE> is publication date for a type of vulnerability to <PROGRAM>.
Wait 2-4 weeks to publish a working exploit.
Don't notify bodies who don't have their own custom code that must be fixed, but happen to pay you enough money.
The latter is actually what happened in this case and it's immensely frustrating.
[1] https://datacentersupport.lenovo.com/de/de/products/storage/...
Should I immediately update my BIOS (would this also patch my CPU's firmware?) or just wait for mitigations arriving with OS updates?
The current recommendation is "no", since some of the microcode updates are known to cause stability problems. It's better to wait for Intel to fix that first, then make sure your BIOS update has a microcode version without the stability problems, then update. Yes, it's a mess.
> or just wait for mitigations arriving with OS updates?
Even after you update the microcode, you still need a OS update to enable the microcode mitigations, so you have to wait for a OS update anyway.
Note that the microcode update does not have to be delivered through a BIOS update. Linux can also update the microcode during early boot:
grep . /sys/devices/system/cpu/vulnerabilities/*
is there an alternate command for ubuntu?
Here is one of the commands recommended in that thread:
grep -q "cpu_insecure\|cpu_meltdown\|kaiser" /proc/cpuinfo && echo "patched :)" \ || echo "unpatched :("
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Checking count of LFENCE opcodes in kernel: YES
> STATUS: NOT VULNERABLE (114 opcodes found, which is >= 70, heuristic to be improved when official patches become available)
CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigation 1
* Hardware (CPU microcode) support for mitigation
* The SPEC_CTRL MSR is available: YES
* The SPEC_CTRL CPUID feature bit is set: NO
* Kernel support for IBRS: YES
* IBRS enabled for Kernel space: NO
* IBRS enabled for User space: NO
* Mitigation 2
* Kernel compiled with retpoline option: NO
* Kernel compiled with a retpoline-aware compiler: NO
> STATUS: VULNERABLE (IBRS hardware + kernel support OR kernel with retpoline are needed to mitigate the vulnerability)
CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Kernel supports Page Table Isolation (PTI): YES
* PTI enabled and active: YES
* Checking if we're running under Xen PV (64 bits): NO
> STATUS: NOT VULNERABLE (PTI mitigates the vulnerability)https://lwn.net/op/FAQ.lwn#slinks
Where is it appropriate to post a subscriber link?
Almost anywhere. Private mail, messages to project mailing lists, and blog entries are all appropriate. As long as people do not use subscriber links as a way to defeat our attempts to gain subscribers, we are happy to see them shared.
> As always, [lwn.net covers the technical details](https://lwn.net/Articles/744287/) about the latest state of the kernel patches to resolve the Spectre issues, so please go read that to find out that type of information.
The non-subscriber link in the blog entry results in a paywall. I think LWN would prefer that Greg used a subscriber link in this case. You may be being downvoted because people think you're trying to post a paywall workaround, but I agree that this would have been an appropriate place to use a subscriber link.
So if someone decides they want to use a somewhat more proactive way of encouraging people to subscribe to LWN, personally I think it's appropriate and its his choice. Just as someone who wants to post a subscriber link to Hacker News is also appropriate. (But you should seriously think about subscribing, even at the Starving Hacker level, if at all possible. I subscribe at the highest tier even though my company has a corporate subscription, Just Because.)
https://lwn.net/SubscriberLink/744287/ec63c9aa1fe236c7/
Please get your company to subscribe to LWN. It's an amazing resource and needs your support.
LWN has tons of great content and deserves any support they can get.
But I strongly suggest that in this case, there shouldn't be a paywall. The current state of Meltdown/Spectre fixes is of high public interest. And people who are normally not into kernel development should be able to read this information without having to pay (or find workarounds).