Why Xen Wasn't Hit by RETBleed on Intel CPUs
xcp-ng.org
xcp-ng.org
It also vindicates the naming of "Spectre", it most definitely keeps showing up.
The article reads a bit too much like "I told you so", but ultimately the author is correct to say it.
Meh. It likely just points out that Xen is a much more limited software environment with much less dependence on indirect branching[1]. There are environments where IBRS has high cost and ones where it doesn't. Linux is in the former category.
Xen also has the advantage of being a hypervisor, meaning that if all they do is expose IBRS to the guest, they can (somewhat cleverly) claim that any resulting vulnerabilities are the fault of the guest software not implementing them. Linux exposes a Unix userspace, and no one told userspace apps they need to use speculation barriers.
Really this article is mostly just marketing. It's a win for Xen, sure, and they should crow about it. But we should recognize crowing vs. genuine security analysis, too.
[1] Vs. say, Linux, which has an extremely robust polymorphic device/bus/probe model where all the methods are function pointers.
Frankly, Xen's work here was not at all impressive; they just applied a fix that Intel told everone to apply. The fact that this is a differentiating thing for them to market with is an indictment of everyone who didn't apply it, and the industry conditions that led to them.
[0] In fairness, the reason we are in this mesz is that said CPU manufacturer has been releasing broken products in the name of speed themselves.
That is not a fair characterization. There are endless mountains of theoretical vulnerabilities[1], and no one (certainly not including Xen) tries to mitigate them all blindly. The dwm post linked in the article explicitly says he's not losing sleep over the issue. Everyone (yes, likely including Xen) believed in good faith that this was not practically exploitable.
> Frankly, Xen's work here was not at all impressive; they just applied a fix that Intel told everone to apply.
And this seems like a misunderstanding too. My gathering from the linked article is that Xen virtualized the barrier mechanism such that the job could be farmed out to guest OSes. Someone running an unpatched Linux under Xen (which is, what, 90+% of the worldwide cloud?) is still vulnerable. But "Xen" is not, which seems maybe less impactful than the marketing being presented would have you believe.
[1] Rowhammer says hi.
All hypervisors do that, including KVM. The difference is that because Xen has to let the guest control the speculation control MSRs, it has to read and write the MSR anyway on every guest<->host context switch. Using IBRS in Xen comes essentially for free.
Linux on the other hand does not have to access the speculation control MSR on every userspace<->kernel context switch, and doing so would have had a bigger performance impact than retpolines. Therefore it took a different approach.
Now the performance impact wasn't that bad on Skylake and it probably would have been good to use IBRS on those processors. FWIW very old versions of RHEL (6 and 7) in fact did use IBRS instead of retpoline because we had little time (there were less than two months from the time the team was put together to the time we had to have something ready to be shipped to customers) and it even took days to read people on the issue because of how secret it was. So we didn't want to put the compiler update on the critical path.
I mean, not nobody. That's sort of the raison d'etre of OpenBSD.
We're talking about a distro that wasn't affected by the latest round of speculation vulnerabilities in AMD's SMT implementation because as soon as they heard about Spectre/Meltdown they immediately realized that SMT was gonna be a giant pile of sidechannels and disabled it on all processors, even the ones that were believed safe at the time. They take "defensive engineering" extremely seriously and will mitigate anything that seems plausible.
That was controversial at the time (extreme performance cost! and AMD isn't affected so why do they have to suffer!?) and they ended up being right, there were more vulnerabilities to come based on SMT leaking data to the other thread.
Nobody mitigates implausible/theoretical ideas that don't seem likely to work, but, a good software engineer certainly should be mitigating things that seem like reasonably feasible extensions of existing attacks, and hardening their environments in general to mitigate the impact if something should pop up. That's not extraordinary foresight, that's just part of the job.
Linus's decision did not follow good engineering practices, and there are examples of other OSs and distros that did do it properly. Xen may or may not not be one of them, it's certainly possible to accidentally fall into a safe path (as AMD likely did on Meltdown, given the broad multi-vendor scope of the vuln), or the "right path" could simply have been easy for them to take, but nobody should be defending Linus on the basis of "nobody could have known". The decisions he made were unsafe and incompatible with a defensive-engineering mindset, and he was told this at the time.
Linus's "why are we doing all this over a handful of broken intel processors" mindset is exactly the trap that OpenBSD avoided falling into. They knew it wasn't just going to be just a handful of broken intel processors, SMT is fundamentally a shared resource and once they saw the basis of Spectre-style sidechannels they knew SMT was gonna be a steady drip-drip-drip of vulnerabilities across all architectures. That was very foreseeable, when I saw the OpenBSD thing at the time it was like "yeah, probably gonna end up being a good call...".
For another "yeah, probably gonna be a problem down the road": KPTI really needs to be enabled-by-default on AMD processors. The Prefetch+TLB attack is still un-mitigated in hardware and AMD relies on KPTI for protection, but still recommends it be disabled by default for performance reasons. The data bleed rate is faster than Meltdown and it's really past time to turn it on by default regardless of what it does to AMD's benchmark numbers. It should have been on-by-default in the first place, and now it's actually got demonstrated exploits leaking kernel memory. Another risky, non-defensive call from the Linux tech-leads.
As bad as Intel's record there was, it's hard to really single it out either. It certainly seems as if the whole industry—CPU manufacturers, integrators, academics, kernel devs, the lot—simply agreed not to notice that this category of vulnerability existed until the moment it was fully impossible to ignore.
But, hey, those Intel/AMD chips sure are cheap! And, let's be honest, that's all that anybody cares about.
We should be slightly careful — while I can't deny that there's a small element of "ITYS" about that tweet, should it really count if I then left it up to Intel to follow up?
I whined that we hadn't shown that it was safe. The credit really does need to go to the RETbleed folks who put in the work to truly demonstrate that it wasn't.
What is the performance penalty for this thing? Was performance the reason for Linux's refusal?
I get the sense that IBRS wasn't used because Linus 1) didn't want to take on the responsibility of fixing something that needs to be fixed in hardware by Intel and 2) didn't want to help Intel avoid responsibility for shipping insecure hardware. The same way he's been antagonistic towards nVidia for being bad participants.
Whatever is the saying about good intentions, you can argue he ended up doing Intel a favour here, by shipping patch that took away only a couple % of performance. Surely, it would've hurt them more, reputation-wise, if default workaround took larger chunk of CPU speed.
It also seems like the changes would've added an amount of technical debt that wasn't acceptable to him ("garbage MSR writes" in and out of the kernel). I could understand why he'd want to avoid a bad solution (that would also have to be maintained long-term) just because Intel is unwilling to fix it on their end.
Xen can implement things like a CPU scheduler exclusively focused on VMs, while KVM has to deal with the normal Linux scheduler for processes.
Xen can do advanced defense-in-depth techniques like driver domains -- something impossible to implement on KVM.
Xen has a mature security response process; if you're a cloud provider, or ship anything with Xen inside of it, you can be notified of security issues typically two weeks before the public disclosure; and we're quite thorough about what we issue security alerts for. For KVM, you just have to hope that your functionality is worth issuing a CVE about, and unless you're a distro, you're only going to be told after it's been made public.
Xen is a microkernel, so you can run it on tiny embedded devices for which Linux / KVM would be too big. Xen is small enough that it's actually feasible to do Functional Safety Certification on it.
That's why Xen is still used by QubesOS, the NSA, and various defense contractors; why a number of cloud providers (including say, Ghandi.net) use Xen; why Xilinx has their own Xen distribution; and why Xen is in the reference implementation for ARM's automotive stack -- in addition to being officially supported by SUSE, and being the driving engine behind Citrix Hypervisor and XCP-ng.
So it's basically a niche thing, mostly used by those who already had it/know it (VMware vSphere is going down that road as well, it's basically legacy today).
Regards,
Xen.
To name one example, nested virtualization support is not only hopelessly broken, it's MORE broken in recent releases than it was a decade ago. You can see right here how the feature kept getting broken by every other release until nothing worked anymore: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xe....
And Xen is literally the only virtualizer out there that does not support nested virtualization, which is a rather critical feature since many (dev) stacks assume one has hardware virtualization, and Windows is going to require it sooner than later.
Regarding nested virt, you are mostly right: it's only "working-ish" for basic things, but indeed, it's broken when you start to use anything heavy in your nested VM. The main reason nobody fixed it is because it's not really used: as any other open source project, you find what you need if you contribute. Obviously, as soon someone will need this and willing to contribute, it will change :)
The implementation is split between KVM and their proprietary equivalent of QEMU.
I do still want to flesh out the Xen guest support in Rust-VMM at some point.
Even if it is a custom hypervisor, AWS likely derived it from Xen and sponsors the project to continue doing so.
AWS does continue to support older instance types, however, but only legacy workloads are using them. They actually now even run the XEN based legacy instance types on nitro. https://perspectives.mvdirona.com/2021/11/xen-on-nitro-aws-n...
I'm not a native speaker, but is this an acceptable written English? The part starting with "versus" seems out of place.
> Mostly because people didn't believe that it was possible to exploit the retpoline limitations, it was somehow logical -but risky- to refuse to implement a mitigation that would slow you down.
In the version you quoted, I'd call it "clumsy" due to a lack of logical comparison from the left- and right-operands of "versus". I could understand what the author was getting at, however.
Unrelated but I am getting so sick of this cliche.
Thinking of more insightful ways to explain why something is challenging is hard.
Maybe it wasn't malicious, maybe it was cosmic rays.
Thanks for the great article.