Critical Xen bug in PV memory virtualization code
raw.githubusercontent.com
raw.githubusercontent.com
[0]: https://aws.amazon.com/security/security-bulletins/XSAsecuri...
This guest escape vulnerability (http://xenbits.xen.org/xsa/advisory-148.html) has been present since 2008 and was found by Alibaba, which recently joined the Xen advisory board, http://www.xenproject.org/about/in-the-news/192-xen-project-...
Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what).
Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.
> Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update the vast majority of our older hardware for this Xen Security Advisory.
https://aws.amazon.com/premiumsupport/maintenance-2015-03/
There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.
Luckily, many people also converted to HVM over PV already. IIRC it was largely PV instances in EC2 classic... but I can't be sure.
Regardless of the virtualization technology, you need to keep on top of disclosures. Any public provider running xen should be on the disclosure list by now http://www.xenproject.org/security-policy.html
| The code to validate level 2 page table entries is bypassed when
| certain conditions are satisfied. This means that a PV guest can
| create writeable mappings using super page mappings.
|
| Such writeable mappings can violate Xen intended invariants for pages
| which Xen is supposed to keep read-only.
"The above is a political way of stating the bug is a very critical one. Probably the worst we have seen affecting the Xen hypervisor, ever. Sadly.
Admittedly this is subtle bug, because there is no buggy code that could be spotted immediately ... On the other hand, it is really shocking that such a bug has been lurking in the core of the hypervisor for so many years. In our opinion the Xen project should rethink their coding guidelines and try to come up with practices and perhaps additional mechanisms that would not let similar flaws to plague the hypervisor ever again (assert-like mechanisms perhaps?). Otherwise the whole project makes no sense, at least to those who would like to use Xen for security-sensitive work.
Specifically, it worries us that, in the last 7 years (i.e. all the time when the bug was sitting there having a good time) so much engineering and development effort has been put into adding all sorts of new features and whatnots, yet no serious effort to improve Xen security effectively. Because there have been, of course, many more security bugs found in Xen over the last years (as the numbering of this XSA suggests)... the bugs in Xen are being found regularly, and this is no good news. For a type-1 hypervisor of the age and maturity of Xen, this simply should not be happening. If it does, it suggests the development process is not prioritizing security."
Linode was actually my failover target for some of those clouds to weather the reboots. They offer both Xen and KVM based instances. If you're still running Xen, you can live migrate your instance to KVM and make all your new Linodes default to KVM via a toggle in your account settings.
I know it is within their right to write things as they please. But seriously, ranting like this is not very constructive and doesn't move things forward. If Qubes thinks the practices in Xen community are bad, why don't they start a conversation on Xen development mailing list?
(edited: typo)
I don't know much about Xen, so perhaps someone else can chime in on this. This exploit says it's for PV, and not for HVM.
But Xen can apparently run both at the same time. Does this mean even if you were using HVM, and Xen had other guests using PV, you were still exploitable?
It's not like Xen HVMs have a better security story than PVMs. The paravirtualization code should probably be more heavily audited than it is already.
You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.
Instead, "nl2e" should have a data type preventing such a direct copy, and only allowing to test single bits.
The code should then be written to copy bits one by one (for bits where it is appropriate), with a comment for each bit stating why it is safe to copy them.
The fact that this is not the case means that none of the other code in Xen can really be trusted to be bug-free, and there is probably no way to fix that without starting over or doing an equivalent amount of rewriting work.
... I'll say a bit more ... I think it's sad that it's not practical anymore to write really high quality code. And most computer security researchers aren't interested in that anymore either, because it's not possible to force all other developers to write high quality code (and if you do you're called "mean" "alienating" etc.) So all they/we try to do is find new layers to contain all the bad code that has been and will be written.
Can we really be surprised when those layers, particularly if they are popular because they came out first and have lots of features, are also low-quality code? Is there anywhere a foot can be put down?
Also, openbsd is another good example of a project with code with a very very low bug density.
Not the space shuttle, but if you think things that go to space (and land on the goddamned Moon) are free of bugs then you are very sorely mistaken.
[1] http://www.fastcompany.com/28121/they-write-right-stuff/
But it would be especially important to have SOMETHING that can run multiple applications while being 100% sure that it is enforcing security between them.
So far everything has been a total disaster, with browsers getting completely owned every year at Pwn2Own, conventional kernels being effectively expected to be locally exploitable at all times (which means all mobile sandboxes are broken as well), and hypervisors where Xen seems to be the most secure of the ones that are mature, and is still terrible.
No, not even if you're counting in binary.
I suspect hiring this company would make even a large budget wilt a bit. Just a bit :P
Of course, the velocity of development is very low, but that's because they want developers who can wrap their head around formal proofs.
Heck, even without a type system, the problem is that the check is backwards. There shouldn't be a mask of flags that the guest isn't allowed to set, the flags should have been &ed with a mask that says which flags they are allowed to set, that way any new flag would have been disallowed by default.
It's nice having a multi-region / multi-provider setup otherwise the typical Rackspace reboot window (much like UPS deliveries or the cable guy) would've been a pain in the ass.
Deleted comment
It's prudent to consider all data on ec2 servers compromised and act accordingly.
We really need a properly written open source hypervisor: entities using secure hypervisors commercially like Amazon AWS should fund the development of one.
Xen was actually written with security in mind, they push as much of the features and driver code out of the hypervisor and onto Dom0 as possible to minimize the attack surface in the hypervisor itself.
Source - The Definitive Guide To Xen - 2007.
Because that should be considered a security hole. Knowing the attack surface is there is, in this context, a security bug in and of itself, because it implies there is already an attack, or an attack is forthcoming.
Unless it's possible to run an unmodified Xen as a guest under Xen, the project is incomplete.
If the kernel itself doesn't just straight up say it's a kernel designed to work under Xen the drivers used to work with the virtualization will give it away. Right down to the version if you care to dig deep enough and compare builds.
http://www.brendangregg.com/blog/2014-05-09/xen-feature-dete...
What makes this bug a huge deal for me, and probably why it's making the news is that it effects fully paravirtualized guests. There have been many, many bugs for HVM guests (and for KVM guests) since the last PV security hole. (March, was it? xsa-123, I think?)
https://rhn.redhat.com/errata/RHSA-2015-1896.html
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456
I could go on and on and on... but the point is that kvm (and xen HVM mode) may have performance advantages in some situations, but xen pv mode has consistently had fewer security vulnerabilities, especially if you use something like pv-grub to load untrusted guest kernels (rather than loading those kernels directly in the dom0)
The biggest problem is the qemu drivers; Honestly, I don't know enough about kvm to know if it was possible, but if you could completely remove the device emulation and force the guest to only use paravirtualized drivers, your security under KVM (or Xen HVM) would be much better, mostly because you'd vastly reduce the amount of hypervisor code the guest interacts with.
But the point here is that all systems have problems, and the xen pv mode has had fewer security holes found it it than most other hypervisors, mostly because there's a lot less code that the guest interfaces with.
s/may/certainly s/some/almost all
It's not a small gap, either. You are basically doubling the number of context switches required for system calls with PV, due to AMD removing CPU protection rings from x86_64, forcing this separation to have to happen in software. You lose out on EPT, so your page table performance suffers in almost all workloads. You cannot take advantage of SR-IOV for NICs or nvme.
PVH might be an answer someday, but for now, there is a pretty massive performance loss. Context switches are important. Memory latency is important. (Nearly) direct access to the hardware is important.
x86 is a nasty enough architecture on its own. x86 plus hypercalls is even nastier and, even more importantly, x86 plus hypercalls is differently nasty.
The current bug is pv-specific. there was another one that effected me early this year[1]... I'm not saying it's bug free or anything, I'm just saying that xen PV has had fewer exploits over the last year (and over the last decade) than HVM mode or KVM.
[1]xsa123 if I remember right; my blog archive is all messed up at the moment. but, for example, xsa-108, which was a big deal, was a non-issue for me because all my stuff is paravirtualized.
Note, the list I'm going off of is here:
It's inferior in basically every way to HVM. CPU performance is worse due to missing protection rings on x86_64, PV drivers for disk and network bring HVM performance to PV for those, SR-IOV for networking makes networking faster, other hardware extension make memory access faster...
THere's very very very very very few workloads that perform better on PV than HVM.
A PV VM would use hypercalls instead of regular system calls, would have more context switches due to the lack of protection rings, etc.