Meltdown and Spectre Linux kernel status | Hacker News Reader

Meltdown and Spectre Linux kernel status

kroah.com

509 pointsdankohn1174 comments

One thing I'll reiterate: as Greg mentioned, the backports to kernels prior to 4.14 are derived from a rather old KAISER version. They do not match what 4.14 and 4.15 do. This has several consequences.

1. They will have bugs. There's a reason PTI was heavily modified from the old KAISER code. They will also tend to diverge from upstream just because the code is so different. This means that the next time low-level x86 changes need to be backported, it'll be a huge mess.

2. There is only minimal upstream support. I, for example, am already largely ignoring two bugs in the backports that aren't in the upstream version. Why? Because I have no affiliation with a distro using an old kernel.

3. Contrary to its marketing, KAISER does not effectively mitigate the old kASLR leaks. PTI very nearly does, and I intend to improve it further once I find some time to do so. I doubt those improvements will get backported to pre-4.14 kernels.

4. At least some versions of "KAISER", on meltdown-affected hardware, expose the kernel stack to userspace. If that's not usable for rooting a box, I'll eat my hat. KPTI doesn't have this problem.

If you can put pressure on your organization or suppliers to update to 4.14 or better, please do so. Red Hat, especially, should seriously consider moving to 4.14 for RHEL 8.

As far as I can tell, the RHEL/CentOS kernel has mitigations not only for Meltdown but also Spectre (using the new MSRs).

Am I right in assuming that Red Hat built their own mitigations, independently from upstream?

It seems like a smallish consortium of poorly coordinated developers built the Spectre mitigation patches. Upstream wasn't involved until a couple days ago.

The patches on RedHat 7.4 have impact on performance and capacity for a Load Balancer type of workload, see https://www.spinics.net/lists/stable/msg209193.html

FYI, Red Hat (and CentOS) already have updated kernels with mitigation.

And that made me asking how Red Hat (and the free derivative CentOS) could have already patched their current kernel (on Jan 4th) while the patch was still being discussed on kernel ML the same day?

I don't see how they could come up with new kernel with mitigation for the three variants, excellent KB articles on the CVEs, tunables flag, preliminary performance test, etc. without having already patches implemented and tested for some time.

Do they have internal kernel devs? And so, do Redhat have another implementation?

(This is my first comment on HN, I hope I'm following adequately the guidelines)

It's a good first comment. Generally: as long as you're not being deliberately mean or derisive, you're doing fine by the guidelines. Welcome!

Kernel ml is public - embargoed/sensitive issues are typically disclosed on closed lists - and discussed internally. RedHat does have many devs that work on the kernel, as does other distros.

Typically, a sensitive issue will be disclosed privately, and devs for various groups will share progress privately. Then ideally, all distros will have patches and documentation ready on the agreed date of public disclosure.

See:

http://oss-security.openwall.org/wiki/mailing-lists/distros

The bug was disclosed to various organizations in early november so work was going for some time. And yes, Red Hat makes significant contributions to the Linux kernel. Red Hat maintains their own version of the kernel and integrating those changes back to the mainstream version can take time to reach consensus.

There's such a thing called: 'embargo period', wherein CVEs are (in most cases) responsibly disclosed to various parties---including several Linux distributions---much ahead than the general public, to coordinate security errata.

So you can be assured that Red Hat coordinates with upstream. It also has a serious "Upstream First" policy (with sensible exceptions, of course).

As to whether Red Hat has kernel developers, of course it does. :-) Much of Red Hat's popularly began as a Linux distribution. If you're curious, the LWN publishes statistics about who contributes to the Linux kernel, and the most recent one is for the 4.11 development cycle -- https://lwn.net/Articles/720336/

(Disclosure: /me is a Red Hatter.)

It's now called coordinated disclosure, for reasons.

And, correct me if I'm wrong, but apparently calling Linus Torvalds and Greg KH the general public (as far as Spectre and its (poor for now) mitigations are concerned) is the new norm. I'm not sure of what the community will think of that. And of the corporate actors involved in that curious choice.

Again, I am pretty sure that they were disclosed the details. This however is very different from being able to commit patches on the day of the disclosure. For one, neither Linus nor Greg are actually writing the code.

The problem here was that discussion was confined within company barriers (with small occasional exceptions) until last Wednesday. Linux thrives because it is based on reaching consensus among all stakeholders, and not being able to discuss the approach to take with respect to Spectre directly resulted in chaos.

I am very happy with the work that we at Red Hat did on this project (though of course not perfect in any way!) and I am proud that our articles are published openly and are being widely shared among the Linux user community at large. However, I certainly would have preferred to work together with all other vendors on a common solution before. But unfortunately, even if (as is the case with Red Hat) upper management is extremely supportive of upstream collaboration, you do what you can.

I should have used the term 'coordinated disclosure', which probably reflects reality more. The links (with remarks from tptacek) pointed out by JdeBP (thanks) up-thread are a useful read.

I really dislike the term "responsible disclosure" in this context. I might consider it responsible to notify a software vendor about a vulnerability in their software so they have a chance to issue a fix to their customers before the rest of the world finds out. But this situation is different. These vulnerabilities affect everyone, and only a special few were allowed to prepare in advance. That's just preferential disclosure.

The Register had some quite pointed remarks on the use of the word "responsible" by Intel in its press releases.

* http://www.theregister.co.uk/2018/01/04/intel_meltdown_spect...

* https://news.ycombinator.com/item?id=16043952

> These vulnerabilities affect everyone, and only a special few were allowed to prepare in advance. That's just preferential disclosure.

In this particular case, most of the fix had to be done in the operating system (the new microcode only enabled extra functionality needed by part of the operating system fixes), so it makes sense that operating system developers were allowed (and required) to prepare in advance. The three most relevant operating systems are Windows, OSX, and Linux; for Linux, one of the most important distributors is Red Hat. That gives two of the groups which were notified in advance: hardware (Intel, AMD, ARM) and operating systems (Microsoft, Apple, Red Hat, a few others).

Ubuntu and BSD still have no fixes. It indeed seems like a preferrential disclosure. Plus no 2nd-tier cloud providers like DO were notified.

Canonical said they were told in November.

It will be pretty unfortunate if it turns out that the projects that maintain a kernel (FreeBSD, various others) only received notification at Christmas, while various Linux distros (who have to deal with packaging, release, QA but not developing their own kernel patch since that comes from upstream) got a long warning period. It seems that way... Looking forward to reading about how this played out when the dust settles.

There is basically no independent upstream (other than a handful of people). Essentially all the kernel developers work for the companies that have distros plus organizations like Intel, Qualcomm, etc. that do a lot of device enablement.

I agree in general. In this particular case, though, a good fraction of the work was done by me and tglx. I'm independent. Tglx is sort of independent.

This is not to diminish the work done upstream by less independent people. Dave Hansen, in particular, is the one who actually got the code to function.

There was a post to OpenBSD- tech list telling no BSDs were told anything. And a blog post from Canonical says that the patch will be available 9th January (IIRC).

OpenBSD didn't respect the embargo a few months ago with the wifi issue, no surprise that they weren't told about this up front.

Yep, FreeBSD got notified a week before Christmas: https://www.freebsd.org/news/newsflash.html#event20180104:01

The Spectre mitigation is expected soon but there is no ETA on Meltdown yet: https://www.reddit.com/r/freebsd/comments/7och5a/freebsd_was...

As others say, OpenBSD indeed violated embargoes a few times.

The lay public, aka everyone, isn't considered to be experts in this domain to address the issues. Hence, they disclosed to top tier OS distributions and partners.

It's similar to the Recalls of Takanas airbags in this regard. The manufacturers are informed and they tell us the consumer to bring in our cars. I'm not qualified to replace an explosive airbag, even though I drive a car daily as a lay person.

There are other OSes and affected companies beyond the ones who were chosen to receive advance warning.

Imagine how it would work out in your airbag analogy if only Toyota and GM knew, and they withheld information for months while they worked on their own fixes, meanwhile the public remains in danger and other automakers have no chance to implement their own recall plans.

Can you mention which OS didn't get early enough warning? I'm sure there is some niche ones which may miss out but they are usually not the target of hacking either. And many of the derivative Linux distributions don't do active kernel development and just compile the stock kernel or derive from Red Hat, Ubuntu or others. The more people that are informed early, the more likely it will leak out early.

My particular interest is in OpenBSD, which was not included.

There is some drama over OpenBSD and KRACK that may have been why they didn't get told. But that is just pure speculation.

I finally found the changelog for the patch that Red Hat applied to their kernel: https://git.centos.org/patch/rpms!kernel.git/6202b1a49906b21... (scroll to mid)

I think we can say that Red Hat does have its own implementation for now.

edit : formatting

> As for how this was all handled by the companies involved, well this could be described as a textbook example of how NOT to interact with the Linux kernel community properly. The people and companies involved know what happened, and I’m sure it will all come out eventually

I'm looking forward to that story.

I'll eat my hat if it isn't some version of "here is a giant patch set we never worked through the chain nor asked about at all please make it your problem to mainline it immediately, followed by, well, a good old fashioned LKML go fsck yourself."

What I'd like to know is how effective are these OS updates (both Linux and Windows) without the associated firmware updates through microcode or BIOS/UEFI flashing. My system is a few years old and I don't expect the OEM to release BIOS/UEFI updates for this model.

Will the OS/microcode update still at least partially protect me or will I have to be super-paranoid about apps and javascript for the remainder of this machine's life?

BTW, my machine is just a normal desktop/laptop, so no server stuff running or expected.

I share your concern. Based on my admittedly loose understanding, although the Spectre vulnerability is "more difficult" to exploit, I suspect it's a matter of time before exploits become more common. And the Spectre vulnerability is the one that requires a BIOS/firmware update.

So that's a bit alarming for me since at least one of my several workstations is using a 5+ year old desktop board (an Intel board, ironically) that has been end-of-lifed according to its BIOS downloads support page. Meanwhile, some of the bigger server motherboard vendors (e.g., SuperMicro) are not yet prepared to supply firmware updates for recent motherboards.

Recognizing that the operating system vendors/maintainers are not responsible for our systems' BIOS and hardware, the flippant advice to "talk with your device's vendor" gives me a sinking feeling. (E.g., Microsoft: "Consult with the device manufacturer about the firmware version that has the appropriate update for your CPU.") As a technical person, I am going to find it difficult to bring all of my hardware—especially the older hardware—up to snuff for coping with these vulnerabilities. I fear for people who are less technically inclined.

>And the Spectre vulnerability is the one that requires a BIOS/firmware update.

One of the Spectre variants. The other isn't mitigated by it and will require an lfence on your sensitive areas of apps.

This is a key detail that gets glossed over too much for my liking.

No OS or Firmware update will fully fix Spectre.

All applications that interpret untrusted code (such as Web browsers) also have to be patched.

https://gist.github.com/woachk/2f86755260f2fee1baf71c90cd653...

Buy stock in Gigabyte, MSI, etc.

Microcode updates can be applied on boot through the BIOS/EFI firmware, but also by the operating system. Linux (I don't know about Windows) has facilities to update the microcode on boot:

https://wiki.archlinux.org/index.php/microcode

https://wiki.debian.org/Microcode

Also, directly from Intel:

While the regular approach to getting this microcode update is via a BIOS update, Intel realizes that this can be an administrative hassle. The Linux operating system has a mechanism to update the microcode after booting. For example, this file will be used by the operating system mechanism if the file is placed in the /etc/firmware directory of the Linux system.

Source: https://downloadcenter.intel.com/download/27337/Linux-Proces...

Apple probably delivers them through EFI updates, but of course, getting updated EFI firmware from Apple was never a problem.

Only kinda effective. Retpoline on Skylake and later isn't completely reliable as a mitigation for one of the spectre variants, and IBRS is one of the processor features that will supposedly correct that.

You'll still be vulnerable, it'll just add another layer of difficulty to the exploit.

The microcode updates that people have been mentioning are not updates to your motherboard's firmware, EFI or otherwise. They are updates to the code that runs inside your central processor chip, the so-called microcode, that does the work of understanding and enacting processor instructions (in all programs, from the programs in your firmware to the programs that you download and run from the WWW).

Firmware updates are largely irrelevant to this issue, only being involved in the sense that one way to perform microcode updates is for your machine's firmware to upload the new microcode image file. But that is just one way for that to be done; your operating system can do it, too.

* http://inertiawar.com/microcode/

* https://news.ycombinator.com/item?id=16081366

* https://newsroom.intel.com/wp-content/uploads/sites/11/2018/... (https://news.ycombinator.com/item?id=16079910)

As I understand the microcode updates get installed by Linux distros and Windows by updates. Microcodes are loaded by the CPU at every boot.

The question is, how can one check if the CPU already got the microcode update?

Can a OS running inside a VM (over Intel VT ring-1) patch the CPU's microcode? (e.g. Linux host, and Win guest patches CPU's microcode) Nothing seems impossible anymore.

AFAIK on Intel it's not possible to load a microcode update over the top of one already loaded by your firmware, so if the firmware shipped an older update, the firmware really needs to be updated. (Please someone correct me if this isn't the case -- I can't even remember where I read this)

I don't think this is correct at all. I've personally seen the contrary happening.

EDIT: however for some rare features, it is preferable to have them loaded by the firmware. I don't know if this is the case here.

Nope, the OS or hypervisor (but not a guest) can freely update whenever they like, as long as all the updates are to newer versions. For updates that modify visible CPU features, like these, it's helpful to apply them before boot. Linux has a mechanism to apply "early" updates from initramfs that effectively function before the kernel boots. I don't think other OSes can do that.

Windows, at least for right now, does not seem to be updating CPU microcode during boot. If you run the PowerShell Get-SpeculationControl script that gives you the status of the updates it states that the CVE-2017-5715 fix requires hardware support. If you don't have it you are told to get a BIOS update from your OEM. WIthout it, the status for CVE-2017-5715 is listed as not enabled due to missing hardware support.

Of all of my computers only one has been issued a BIOS update so far. I ran the PowerShell script before and after installing the BIOS. Installing the BIOS flipped the status of the fix for CVE-2017-5715 to on.

I also ran the free HwInfo utility that will show the microcode revision before and after installing the Windows update and before and after installing the BIOS. It only ever changed with the BIOS update.

I hope this changes and Windows does start updating CPU microcode. I also have a few older motherboard's that aren't likely to ever see a BIOS update again.

The situation with Windows NT is slightly tricky, and underreported. I have pointed to this once already on this page, but it bears repeating.

* https://news.ycombinator.com/item?id=16076660

This appears to be my experience too. I'm checking a few VMs and they are showing as missing hw support.

Do note that Microsoft appears to have updated their doc in the last day or so. They are now saying that three registry settings need to be set (instead of 2 previously).

  reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f
  reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f
  reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

Some quick Googling seems to indicate that Windows has the ability to update microcode.

http://forum.notebookreview.com/threads/how-to-update-microc...

Maybe this will happen with a later update from Microsoft as I'm doubtful that BIOS updates will happen with at least one of my systems.

MinVmVersionForCpuBasedMitigations is for VM/Hyper-V: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-w...

The microcode updates can typically be stored and loaded by the motherboard fw, though, and some OSes don't even load their own version.

VMware distributes a driver[1] one can install to patch the microcode on Windows at runtime. If these updates need to be recognized before OS boot, this may not mitigate the dangers. But in my test lab, HWiNFO64 shows the current microcode version.

The Intel[2] and AMD[3] microcode can be downloaded but I can't vouch for how many Meltdown/Spectre mitigations these versions contain. The Intel one is dated 2017-11-17.

[1] https://labs.vmware.com/flings/vmware-cpu-microcode-update-d...

[2] https://downloadcenter.intel.com/download/27337/Linux-Proces...

[3] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...

If you're running a Windows machine, I've no idea how it handles the microcode updates.

For LINUX (and probably BSD/etc users):

https://wiki.archlinux.org/index.php/microcode#Enabling_Inte...

Your distribution probably has a firmware package that has this bootloader stub. It applies the latest microcode updates just before your OS actually begins starting up (but for Linux after the Kernel has initialized things and is preparing to hand off to the initrd).

Yes, this is something that needs to happen at every boot; they don't get burned in to the CPU.

I downloaded the microcode updates from Intel, but hesitate to apply them because one of the expected directories does not exist on Manjaro, so I'm out of my element on how supported the update would be. Do you happen to know if in general microcode updates are distributed in packages? It seems that's probably the case, and if so, I'm fine waiting.

I use AWS instances (multi-tenant). I understand that by now AWS hypervisors have been patched.

Does that fully protect my unpatched AWS instances from this CPU-level issues?

If not, is there any way to protect my AWS instance from a rogue unpatched attacker instance running on the same hypervisor?

In other words, with the current CPUs deployed at AWS, will it be possible for an attacker to simply launch an unpatched instance to steal data from other instances (patched or not) running on the same AWS hypervisor?

I hope not, because the cloud multi-tenant model would be effectively dead until AWS hosts with unaffected CPUs are available.

In general, the multi-tenant security model does not rely on a malicious tenant being limited to any OS or patch (they can and always will be able to run whatever code they want including OS, and it's designed to still be secure). The cross-VM attacks we've seen actually go through the hypervisor, and since that is patched you should be fine if you're patched.

No. The "master" software fix to Meltdown keeps userland and the kernel from trivially sharing kernel memory pages. If your guest kernel doesn't have this fix applied, then your guest userland shares pages with your guest kernel, and guest processes can dump kernel memory.

In theory it should be possible for a hyper visor to retrofit a fix into a guest, but it's messy and I doubt anyone will ever do it. Could be fun though.

Quick sketch: disable EPT and go back to shadow paging. Maintain a third page table with any kernel pages unmapped. Invisibly swap between on syscalls.

This sounds really slow to me. What is the overhead of "disable EPT and go back to shadow paging", or what was the performance win of EPT?

What bothers me is the lack of confirmation on the microcode updates (other than 'soon') – apparently they're out there[1], but Intel's own package has yet to be updated[2]. The 20171215 update carried by some distros only seems to cover HSX/BDX/SKX, so does that mean regular HSW/BDW/SKL users are screwed, are they covered by the 20171117 update, or are the new microcodes simply not ready for all models yet?

Of course, the microcode updates are meaningless without the corresponding kernel patches, but apparently only RHEL and SLES were deemed worthy of receiving those ahead of time, having already rolled them out while Linus and co are left scrambling to integrate the IBRS and retpoline code dumps after the fact.

[1] https://tracker.debian.org/news/899110 [2] https://downloadcenter.intel.com/download/27337/Linux-Proces...

Just to add to that, it looks like some manufacturers have already pushed out the new microcode alongside the recent ME fixes.

I flashed my Skylake laptop a few days ago, and now I'm on microcode rev 0xc2 (versus 0xba in the latest Intel tarball). CPUID output suggests IBRS support (according to [1]).

[1] https://patchwork.kernel.org/patch/10147547/

Reading between the lines, it seems this is a pretty colossal fuckup by Intel; both in terms of the bug but especially how it was handled.

You do not need to read in between the lines to learn that; several commentators have come right out and said it.

* http://lists.dragonflybsd.org/pipermail/users/2018-January/3... (https://news.ycombinator.com/item?id=16084641)

* https://medium.com/@frankycaron/this-week-in-words-the-langu... (https://news.ycombinator.com/item?id=16075588)

* http://www.theregister.co.uk/2018/01/04/intel_meltdown_spect...

One of them you might have expected to use the word "fuck", but actually did not. (-:

* https://lkml.org/lkml/2018/1/3/797 (https://news.ycombinator.com/item?id=16066968)

In part, this was down to the cat being let out of the bag early.

* https://news.ycombinator.com/item?id=16080918

* https://news.ycombinator.com/item?id=16068510

If those 4 extra days of embargo would not have caused a total PR disasters on Intel's side will eat my socks. This wouldn't have become any better given another week of time.

The fact that Intel is rumored to have know about this since June lends credence to your belief, however if that week would have let OS vendors have updates ready, and cloud providers already patched, then it would be less of a disaster. There's a certain amount of flailing about when details are released earlier than expected, and vendors don't have a answers or instructions prepared.

Reading between the lines would be figuring out who at or around Intel profits from this chaos. There is certainly someone, but since everybody is just focussed on the technical issues, whoever it is will probably throw a party this weekend.

No, one does not need to read in between the lines for that, either. One can simply read the lines. Far from everyone being "just focussed on the technical issues", the questions over the share dealings have made the news in the papers, from The Australian to The Irish Times.

Insider trading happens all the time. But usually you don't read about it in the news. So that the Intel CEO's insider trading got him in trouble shows there's someone fighting against him. Thus, no that's not the full story. I know it's hard to accept but the official scape goat is rarely the trouble maker. Or do you believe that the 2008 crisis was just the doing of that one banker that got jailed?

The 2008 crisis was an emergent systemic risk. (Similar to Spectre) But Meltdown is Intel's fuckup, and the insider trading is on Krzanich. And naturally anyone is happy to point that out, even if they sort of fucked up by implementing speculative execution and thus allowed Spectre on their platform.

Or it can be simply someone that noticed this unusual sale, and saw the circumstances ripe for a bit of naked shorting.

Right. Shorting is also an option. Haven't thought about it. It would also be insider trading, though, and it's not in the news that someone shorted Intel stock big time.

From my cursory reading I understand it is a cleverly orchestrated timing attack.

In other words, if something would need 500 picoseconds you have bit 1, if it is 250 picoseconds instead it is bit 0 (numbers pulled out of thin air).

This is made possible because processors execute the read speculatively even if it is actually forbidden. This read causes a cache hit. Of course the read is never brought into effect because first it is forbidden and second it is in a branch which will never be executed. At the time of the speculative read the CPU seems not to have enough information to know not to execute that read. Even speculatively.

And that speculative read has a side effect on the cache which is measured by the exploit.

Of course the read is never made visible to the process because later the CPU knows not to bring it into effect. However then it is too late: there is measurable difference in caching times.

Did I understand the way of the attack correctly?

Timing the L1 cache response for speculatively fetched data that one could not normally access is just one of the problems. It's the one that it is easiest to explain, and it's also the one that the few operating system writers who were told about this tackled first. Hence it is the one that is getting the focus in many discussions.

* https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulne... (https://news.ycombinator.com/item?id=16080002)

* https://dev.to/isaacandsuch/how-meltdown-works-28j2 (https://news.ycombinator.com/item?id=16085592)

One of the other problems is tricking the branch predictor into speculatively jumping into code of one's choosing.

* https://newsroom.intel.com/wp-content/uploads/sites/11/2018/... (https://news.ycombinator.com/item?id=16079910)

> One of the other problems is tricking the branch predictor into speculatively jumping into code of one's choosing.

But to read the data one still must use timing attacks, no?

Yes, you understand it correctly. The bug is really just that cache hits that are caused by speculative execution do not get invalidated. This is technically a bit of a speed-up, but it leaves information behind about the speculative execution that took place.

It is also worth understanding the differences between Spectre and Meltdown, since they are distinct. Spectre refers to reading process memory via measuring cache timing after speculative execution. Meltdown, in conjunction, refers to the fact that Intel CPUs do not verify access rights to a virtual address until after the speculative execution (and thus the cache hit) takes place. Meltdown allows bypassing page permissions so you can read any page mapped in your virtual address space. Spectre is 'limited' to only reading addresses you already have access too.

so the perf hit comes from a mandatory cache invalidation instruction added by the patch ?

No. That would be the ideal situation, but it is not possible to make the CPU do that just from a microcode update, it would require new hardware. So we're left with creating other ways to get around it via software. The perf hit comes from those software fixes.

Meltdown is the most expensive perf wise, but more straight forward to fix. Processes can only access addresses mapped in their address space, so if you in unmap the kernel while a user process is running then they can't read it. This is expensive because every syscall now flushes the TLB due to changing the page table, so page accesses are in general slower.

Spectre is more complicated to fix. One part of the fix is the retpoline hack that basically attempts to defeat the branch predictor through clever code. It looks like the CPUs are also getting microcode update to allow it to disable the branch predictor in some situations.

> If you rely on any other kernel tree other than 4.4, 4.9, or 4.14 right now, and you do not have a distribution supporting you, you are out of luck.

So, if you are running a (still supported) Debian Jessie a simple apt-get upgrade isn't gonna cut it:-(

As the Debian announcement says:

* https://lists.debian.org/debian-security-announce/2018/msg00... (https://news.ycombinator.com/item?id=16076175)

It's not the only operating system where the process is slightly more complex than a kernel update. Windows NT updates require the coöperation of other softwares on one's machine. (-:

* https://news.ycombinator.com/item?id=16076660

I'm not pointing at Debian. I think it is even possible to run Jessie with a current kernel, (which is quite cool) but I haven't tried it. I just upgraded all machines to Stretch and called it a day.

I just did the apt-get update; apt-get upgrade dance in a hurry yesterday and thought I might be good. My post was more a reminder to everyone not be lulled into a false sense of security...

I run (many) jessie systems with 4.9 kernels (still haven't made a 4.14 configuration). No problem at all.

You might want to bite the bullet and upgrade from 4.9, because the patches for 4.9 are not so rock solid: https://news.ycombinator.com/item?id=16087736

The debian security announcement for stretch says

> For the oldstable distribution (jessie), this problem will be fixed in a separate update.

You can check https://security-tracker.debian.org/tracker/CVE-2017-5754 whether the KPTI patch has been released for jessie.

Well, you only need to cover yourself between now and whenever you can get your hands on an AMD chip. That cpu/board swap will be much more work anyway.

Someone please correct me if I'm wrong, but both spectre and meltdown seem to me to be local root exploits, not remote vulns. They can be used to break out of (say) a VM into the host hypervisor, and thence into other VMs running on the same hardware, but cannot be used to break into a machine from outside its hardware perimeter. Is that right?

Essentially yes, the exploits require code running on the machine that’s attacked. However, for example JavaScript runs on the local machine and is a demonstrated attack vector.

It’s also strictly speaking not a privilege escalation, it’s “see things you’re not supposed to.”, such as all sorts of secrets. The attacker does not gain any write or execution privileges, though.

> JavaScript ... is a demonstrated attack vector

OMG, I didn't know that. Thanks.

JS as an attack vector can be effectively mitigated by denying access to high-precision timers (and a few features that can be used to construct high-precision timers). At least Chrome and Firefox are doing this, so just make sure that you are keeping your browser up to date.

If you feel paranoid, you might also want to disable JS by default and only enable by whitelist on any machines that hold particularly sensitive or valuable data.

Keeping JavaScript from constructing high-precision timers is going to be a cat and mouse game.

See the "Fantastic Timers" paper. Link to HN discussion: https://news.ycombinator.com/item?id=16080235

Also: if you don’t have high precision timers, you probably could also go and increase the number of guesses to cancel out noise. After all, this is a statistical side channel attack. This will be fun for quite some time.

Doing that is harder than usual, because the time before rollback is very limited, reducing the amount of cache lines you can pull into the cache and you can measure a given cache line only once.

The attack is already pretty damn slow (1kB/s) in C, if you have to collect a statistically significant sample in JS it might well slow down enough to not really work properly.

It also consumes lots of CPU to scan memory this way, so just be on the lookout for webpages consuming lots of unexpected CPU.

In the age of fancy JavaScript webapps, this seems laughable. 2 or 3 tabs in Chrome is enough to turn my laptop into a space heater.

That describes most websites. I guess noscript might help, but it's a pain to manage the whitelist for that.

Yeah, it's possible to use to bypass same origin policy and read data across origins. Basic idea is that most browsers have multiple origins in the same process, meaning that the same speculative execution + optimistic caching can leak data to js from another origin in the same process.

> JavaScript […] is a demonstrated attack vector

Does that mean that we are soon going to be vulnerable to remote exploits via WASM as well? Do mitigation attempts such as reducing the JavaScript timing resolution aslo apply to WASM?

They do apply to WASM. However, these mitigations are hardly air tight. I'd expect new exploits anyways.

All that these mitigations did was disable all obvious precise timers. All you need for a new attack is a novel way to create a precise timer. And since it’s now known how valueable such a timer would be, I’m certain that minds more clever than me will take a shot at this.

If one of the secrets you see is the contents of a buffer containing a password, it might then be a privilege escalation.

My understanding is that they require execution on the machine and even systems like node are vulnerable(https://react-etc.net/page/javascript-spectre-meltdown-faq) The big issues is that this is really hard to detect because it blends with what would be normal code patterns. I have read that AV will have trouble with this in heuristics but may pick up on specific strains of code that utilize it. Plus this is a whole new class of vulnerability that people have not thought about yet, it will be interesting, can I make your server do X to expose this(think Heartbleed)

Yes, but arbitrary code execution exploits are some of the most commonly found. If your system interacts with the outside world, or is a multi user system, I would consider it prudent to protect against both Meltdown and other Spectre class exploits whenever possible.

With so much going on - is there a way in linux to know whether my system is patched or not?

Similar to the powershell script for Windows?

cat /proc/cpuinfo | grep pti

What is the expected outcome if its patched vs. not?

I am getting:

  fpu_exception	: yes

You should see "pti" on the "flags" line of /proc/cpuinfo. You apparently don't, so you don't have page table isolation.

No, he shouldn’t be seeing that regardless as it doesn’t match the grep regex. I don’t know what, though.

Edit: facepalm. Or should that be foot-in-mouth?

It does match the regex. "pti" is a substring in "exception"

Thanks. I feel stupid, I looked that line up, down, and sideways for a match :/

I was wrong

fpu_excePTIon

I currently have the following system (up-to-date):

  Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-104-generic x86_64)

I ran the following command:

  sudo cat /proc/cpuinfo | grep pti

which returned:

  fpu_exception	: yes

Note the instance of 'pti' in the word 'exception'.

$ dmesg | grep isolation

[ 0.000000] Kernel/User page tables isolation: enabled

A number of ways are listed here [1]

1. With dmesg

dmesg -wH | grep 'page tables isolation'

2. With /proc/cpuinfo

grep cpu_insecure /proc/cpuinfo && echo "Patched" || echo "Unpatched!"

[1] https://askubuntu.com/questions/992137/how-to-check-that-kpt...

Thanks. So apparently, my system:

  Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-104-generic x86_64)

is unpatched! Is it because of LTS version? Most servers run this including mine.

Ubuntu has not released kernel updates for this issue yet.

Ubuntu say they will release a patch on or before the 9th of January.

They got caught out by the embargo being ended early.

Ubuntu’s kernel package updates are pending.

This Ubuntu Wiki page is being updated with relevant information and package updates as they become available:

https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAn...

I like

    dmesg -H | grep 'page tables isolation'

better, because dmesg -wH doesn't return and I suppose the 'page tables isolation' appears during boot.

Note that `dmesg` prints from a fixed-size ring buffer, so if your system has significant other output (e.g. if you create and destroy lots of devices or such), it's very possible the boot-time message about page table isolation will have fallen out of the ring buffer.

If you use journald, it by default saves the kernel's ring buffer to disk, so you can use it to check for that message:

sudo journalctl -b -o cat | grep "page table isolation"

Interestingly I get:

    dmesg -H | grep 'page tables isolation'
    [  +0.000000] Kernel/User page tables isolation: enabled

    grep cpu_insecure /proc/cpuinfo && echo "Patched" || echo "Unpatched!"
    Unpatched!

    cat /proc/cpuinfo | grep pti
    fpu_exception	: yes

    uname -a
    Linux host 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

So page tables isolation seems to be enabled but neither the pti flag nor the cpu_insecure bug is in cpuinfo.

EDIT: Maybe this is because it is Xen guest. Do I need pti on a XEN guest if the host is fully patched?

> grep cpu_insecure /proc/cpuinfo && echo "Patched" || echo "Unpatched!"

That command checks for the "bugs: cpu_insecure" entry in /proc/cpuinfo. However, that line only appears in some of the kernel versions. Recent kernels will have either "cpu_insecure" or "cpu_meltdown" (the name has been changed), while for instance the 3.10 kernel from CentOS 7, which has a backported version of these patches, doesn't even have the "bugs:" field.

And it's that 3.10 kernel which has all the workarounds (both for Spectre and Meltdown), while the more recent kernel has only what's been upstreamed, which so far is only the Meltdown workaround.

It's a mess.

Link to the powershell script for those who need it:

https://blogs.technet.microsoft.com/ralphkyttle/2018/01/05/v...

Based on the remarks I'm wondering if there's a distribution out there that is similar to Ubuntu LTS but only uses LTS kernels.

I'm on CentOS right now and love it, but from what I've understand so far it would be preferable to be on a newer kernel.

I was planning to look at Ubuntu LTS and while they do have updated kernels available they seem to ignore LTS: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

Also, 18.04 will be based on 4.15 which doesn't make sense to me.

Debian stable maybe?

EDIT: I know I can install updated kernels but I'd rather stick with what a core distribution provides if possible

Actually, Ubuntu created the term "LTS" in 2006, for Long Term Support releases. Greg started using the term himself just a couple of years ago.

Re-reading my post it does sound a bit confusing.

What I meant is: Ubuntu LTS seems to not give a damn if the kernel they use is also an LTS release.

Examples would be: 14.04, 16.04 HWE kernels, 18.04 (currently planned to be 4.15).

We applies it to the bare metal DB servers for my.zerotier.com and there is definitely a detectable increase in IO cost. Reducing syscall overhead is going to become even more important.

What about Ubuntu users? Is the update our already?

Not yet - keep an eye on their wiki page for Meltdown/Spectre:

https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAn...

I've been thinking about some of this. A possible design to prevent some of these issues would be to read all protected memory, in speculative execution paths, as dummy all-zero bits value, and not put anything into the cache. I.e. no actual access takes place to any protected area and no breadcrumbs are left in any CPU storage such as a cache. (Then if that path is actually taken, generate the exception).

The whole problem is that the kernel/user protection is just a sham that is not being taken seriously by the CPU implementation. It allows access to take place without a proper mode change. Basically the whole fetch-decode-execute machinery (all of its pipelines, caches, registers and other areeas) should be regarded as untrusted and prevented from accessing or storing anything unauthorized according to the current security state of the CPU.

The kernel/user enforcement by the CPU is a sham (kinda) in Intel CPUs when the data you want to access is in L1 (people are guessing). This is only for the meltdown issue.

Spectre doesn't care about kernel/user at all. So even other CPUs are susceptible. Every CPU that speculates is vulnerable in theory.

Is the clock still ticking on Joyent.com patching their public cloud (illumos, SmartOS)?

https://help.joyent.com/hc/en-us/articles/115015938847-Secur...

    If your Linux systems are running a normal Linux
    distribution, go update your kernel. They should
    all have the updates in them already.

I use Linux Mint and have not seen any recent kernel updates.

Which kernel version is considered safe?

Almost all Linux Mint users "piggy back" on top of the most recent Ubuntu Long Term Support (LTS) release. Currently the most recent Ubuntu LTS is 16.04 and by default it uses the 4.4 Linux kernel. There are patches for the 4.4 kernel from the kernel.org team and the Ubuntu team are testing and integrating these patches. The most recent updates from the Ubuntu team about Meltdown and Spectre is available from this URI: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAn...

So, there is NO kernel update available for Ubuntu 16.04 at this time.

Right - for desktop use though, there are Firefox and Chrome updates with mitigation. JavaScript exploits were the most dangerous desktop scenario.

For servers running Ubuntu, what is the risk, as long as my services don't run arbitrary user uploaded executables? As far as I can tell it is that a different remote code execution exploit can now read the entire memory, possibly leaking secrets. Assuming we have a kernel update in the next few days, I would need to install it immediately and rotate passwords and keys. Should I revoke TLS certs? Is that paranoid?

I think it's naive to think you're completely protected just because code isn't supposed to ever run. It seems as though the simplest and safest piece of mind is to use some extra layers of protection ala SELinux.

This won't stop the memory from being accessed, but it has a better chance of stopping things that can exploit the bug(s) in the first place.

Revoking TLS certs is probably a little bit on the side of paranoia.

I think you're on the right track -- just watch for the kernel update, and rotate passwords plus keys if it's not a hassle.

It’s naive to assume that your system is perfectly secure in preventing unauthorized code from running. But that’s just in general.

He is right, though. It would take two vulnerabilities to pwn him: one allowing remote code execution and then another (Spectre/Meltdown) to gain access to privileged data that shouldn’t be available in that context.

Too many machines put on too many hats. A single (physical) “secure” server should do as little as possible and run as small a codebase as possible. And never run - sandboxes or otherwise - code that isn’t authorized.

We seem to be forgetting that in all this. If you only run code you trust, you are safe. This can only happen if you run in trusted code on your machine. We’ve taken running untrustworthy code in a “sandboxed” environment to mean “not running untrusted code, when it’s totally not the case.

Things are definitely getting lost in the panic here. It's going to take several weeks for everyone to get their head on straight, but yes, PTI is only going to be justified in certain situations, and if you don't allow untrusted code to run on your system (most servers), you will probably be fine with just your host CPU patched because no one will get the chance to run the exploit.

Of course, if an attacker uses a remote execution vulnerability to get into the box with user-constrained permissions, this can be used to read guest memory without concern for user limitation, so in that way it's a long-term pernicious threat that will make local exploitation significantly easier, and security-conscious organizations will still opt to use it despite the fact that they don't run untrusted code. Also, since this would allow them to read all memory in the guest, if you have sensitive stuff like database credentials coming into memory, they could be sniffed without requiring further exploitation.

Also, consider that at present, PTI is disabled by default for AMD chips, based on AMD's assurances that Meltdown does not affect them. If you're running in the cloud and your host is AMD-based, you don't need PTI in either the guest or the host.

They will be available on 2018/01/09:

"Ubuntu users of the 64-bit x86 architecture (aka, amd64) can expect updated kernels by the original January 9, 2018 coordinated release date, and sooner if possible."

https://insights.ubuntu.com/2018/01/04/ubuntu-updates-for-th...

The status of patches can be viewed at the CVE pages listed at https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAn...

4.4.110 is available (and patched) right now from kernel.org . Compile it yourself and you'll be safe(r).

Better to recommend something from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/ (especially the daily is relevant now)

http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/

> I use Linux Mint and have not seen any recent kernel updates.

If you're not aware Linux Mint turns off kernel security updates _off_, you need to check by hand.

That hasn't been true for years.

I'm sorry but "the user has to opt-in during installation" doesn't count.

I always preferred Mint for desktop Linux. I knew they had a poor security record due to their HTTP downloads at one time, but no kernel security updates is news to me.

If you must use desktop Linux, I guess I'll be recommending Ubuntu from now on. In my testing Mint and Ubuntu both work pretty well out of the box on lots of different hardware configurations unlike other distros. But learning this really tips the scale, Ubuntu or bust.

A great advantage of linux is allowing the end user choice of distribution and desktop environment to find their specific needs. Unfortunately, not all choices are good. I wish Mint would go away.

Here's the directly subsequent paragraph:

> However there are lots of systems out there that are not running “normal” Linux distributions for various reasons (rumor has it that it is way more than the “traditional” corporate distros). They rely on the LTS kernel updates, or the normal stable kernel updates, or they are in-house franken-kernels. For those people here’s the status of what is going on regarding all of this mess in the upstream kernels you can use.

Mint is one of those.

A good way Intel could say sorry is to have a new generation hardened cpus at a good price.

Even offering ECC for mere mortals would be a good start. Of course with the option to disable as much of the ME as possible.

So if you're running a ARM64 chromebook, the answer appears to be: get fukt.

"For the 4.4 and 4.9 LTS kernels, odds are these patches will never get merged into them, due to the large number of prerequisite patches required. All of those prerequisite patches have been long merged and tested in the android-common kernels, so I think it is a better idea to just rely on those kernel branches instead of the LTS release for ARM systems at this point in time."

Great. Into the trash it goes.

This just reminded me: when will Ubuntu (and Debian?) fix apt’s broken kernel update process? I have never seen a kernel update - security or otherwise - installed via a normal “apt update; apt upgrade” on any of our machines, it’s always “the following updates have been held back” and then it’s time to manually use dpkg to install the relevant updates.

“apt full-upgrade” will upgrade everything. It may also remove packages to fix higher priority dependencies, though I’ve never seen this happen in real life.

Happened to me yesterday with my Proxmox host. It wanted to install firmware-linux-free, which conflicted with pve-firmware. It decided it wanted to remove pve-firmware which would also completely remove proxmox-ve... Had to upgrade with --no-recommends.

have you tried dist-upgrade?

It's not supposed to install via "apt upgrade", as that command won't pull in new packages or remove packages. Unlike most, kernel packages can co exist so they are all separate packages. Thus need the extra "apt-get dist-upgrade" or "apt full-upgrade" variants as they are new dependencies.

"apt upgrade" (unlike "apt-get upgrade") does pull in new packages, but it won't remove existing ones.

I don't see why "apt upgrade" wouldn't install the new kernel. It's hard to tell what when wrong without knowing any details.

Anyway, if apt decided to hold back a package for some reason, most of the time you should be able to install it with "apt(-get) install". Do not install packages with dpkg unless you know what you're doing.

You need to run dist-upgrade or some other variant that will install new packages to pull in kernel upgrades. The reason is that each new kernel is its own package, depended on by (in the case of Ubuntu for example), the linux-generic package. apt upgrade wont install new packages, dist-upgrade will.

other options include "apt upgrade --with-new-pkgs" or using unattended-upgrades to automatically install them.

--with-new-pkgs is already the default for apt, thus a plain apt upgrade indeed installs new packages.

apt's kernel update process is not broken, you've just somewhat borked your system either by removing the kernel meta package or messing with repository sources/priorities.

Make sure "linux-image-amd64" package is installed for Debian (and whatever equivalent package name for Ubuntu) and "apt upgrade" will install new kernel packages just fine.

(And this is actually not even needed for most kernel updates in Debian because most of the time package name stays same and updated in place, package name is only changed if ABI is changed and requires recompilation of DKMS modules)

The website serves JS and does not serve it over https, and discusses Spectre bug and how to patch it. I know he is the second most important guy on linux, but the irony.

And you probably know much better than the second most important guy on linux what security means on the Internet: securing some javascript script on a random blog.

HTTPS should be expected by now - ISPs keep messing with my unencrypted traffic.

Then you should get yourself a different ISP or a VPN.

Some places have no other choices for ISPs. I don't disagree that a VPN would help but that feels like it's simply dismissing that there's a problem.

This is an extremely easy thing to say for many people around the world.

But for a large number of people - Americans, folks in countries with monopolies or state manipulation of internet traffic - it is not.

Not everyone has a different ISP to choose from. VPNs are a risky proposition and can significantly reduce bandwidth and increase latency.

Anyone that thinks ISPs don't mess with traffic, should check out this malware research

https://www.virusbulletin.com/conference/vb2017/abstracts/la...

It's not about that guy's blog. This causes desensitization to non-HTTPS traffic and when people then actually visit non-HTTPS malicious blog, they get infected. If all "trusted" websites were HTTPS, then whenever there was untrusted access, people will notice it and raise alarm.

On a related note, accessing the site over Chrome will full site isolation can help protect against both bugs: https://support.google.com/faqs/answer/7622138#chrome

I get the irony, but if you could magically visualize the entire internet security threat matrix, this would fall so far down the list and his other work is so high in terms of impact, that it would absolutely no sense for him to take even one minute away from his other activities to address this.

Yes, hence my question. I have tried to setup a HTTPS service and it seems incredibly complicated if you have your own domain name. Even with lets encrypt, if you are using github pages for hosting but your own custom domain, you are out of luck. The point is not that he is not serious about security, point is, it is too hard to get normal security correct. And I am not talking about "cryptography is hard". I am talking about tools which should be easy and standardized. Those are hard.

What question?

I'm out of the loop but I heard it's the white hat hackers who exposed the flaw (probably they were at Google?).

Is the exposition carefully publicized so the flaw is not exploitable by malicious hackers?

Or does Project Zero expose everything, and a malicious hacker can read it and create code that spreads over the internet to harm computers?

I hope it's not the second case because that should cause global panic.

It's closer to the second, but:

1) the vulnerability is local, not directly tied to spreading as malware (but these days placing JavaScript in an ad is easier and possibly more effective than a virus...)

2) there is no such thing as "exposition carefully publicized so the flaw is not exploitable by malicious hackers". Just assume that black hats are as smart as white hats or smarter.

> 2) there is no such thing as "exposition carefully publicized so the flaw is not exploitable by malicious hackers". Just assume that black hats are as smart as white hats or smarter.

You're missing the point. Not doubting the smartness of black hats, white hats likely took their time to discover the flaw. If you make the details public in a controlled manner and then announce the fixes shortly after, you essentially did not give black hats enough time to fill the missing pieces of the public announcement.

For instance, in the extreme case, the statement "we have discovered a flaw at the hardware/CPU level in such and such chips, and we call it meltdown and spectre", it's pretty obvious the black hats would have no clue what it is. (They may have already discovered it on their own, and may have named the flaws something completely different. Even then they wouldn't know if white-hats discovered what they discovered.)

For one thing, the fixes are up in the Linux kernel for anyone to see.

unfortunately the fixes are in an open source project so even if all they knew was that there was a CPU level exploit and that specific code fixed it, black hats would still have a weaponised exploit before most people had patched their systems.

Reply on news.ycombinator.com