Attacking Firecracker: AWS' MicroVM Monitor Written in Rust
graplsecurity.com
graplsecurity.com
Firecracker is a KVM hypervisor, and so a Firecracker VM is a Linux process (running Firecracker). The guest OS sees "physical memory", but that memory is, of course, just mapped pages in the Firecracker process (the "host").
Modern KVM guests talk to their hosts with virtio, which is a common abstraction for a bunch of different device types that consists of queues of shared buffers. Virtio queues are used for network devices, block devices, and, apropos this bug, for vsocks, which are a sort of generic host-guest socket interface (vsock : host/guest :: netlink : user/kernel, except that Netlink is much better specified, and people just do sort of random stuff with vsocks. They're handy.)
The basic deal with managing virtio vsock messages is that the guest is going to fill in and queue buffers on its side expecting the host to read from them, which means that when the host receives them, it needs to dereference pointers into guest memory. Which is not that big of a deal; this is, like, some of the basic functioning of a hypervisor. A running guest has a "regions" of physical memory that correspond to mapped pages in Firecracker on the host side; Firecracker just needs to keep tables of regions and their corresponding (host userland) memory ranges.
This table is usually pretty simple; it's 1 entry long if the VM has less than 3.5G, and 2 entries if more. Unless you're on ARM, in which case it's always 1 entry, and the bug wasn't exploitable.
The only tricky problem here for Firecracker is that we can't trust the guest --- that's the premise of a hypervisor! --- and a guest can try to create fucky messages with pointers into invalid memory, hoping that they'll correspond to invalid memory ranges in the host that Firecracker will deference. And, indeed, in 2019, there was a case where that would happen: if you sent a vsock message, which is a tuple (header, base, size), where:
1. The guest had more than 3.5G of memory, so that Firecracker would have more than one region table entry
2. The base address landed in some valid entry in the table of regions
3. base+size lands in some other valid entry in the table of regions
There are two bugs: first, a validity check on virtio buffers doesn't check to make sure that both base and base+size are in the same, valid region, and second, code that extracts the virtio vsock message does an address check on the buffer address with a size of 1 (in other words, just checking to see if the base address is valid, without respect to the size).
At any rate, because the memory handling code here deals with raw pointers, this was done in Rust `unsafe{}` blocks, and so this bug combination would theoretically let a guest trick Firecracker into writing into host memory outside of a valid guest memory range.
The hitch, which is as far as I know fatal: there's nothing mapped in between regions in x86 Firecracker that you can write to: between a memory region and the no-mans-land memory region outside it, there always happen to be PROT_NONE guard pages†, so an overwrite will simply kill the Firecracker process. Since the attacker here already controls the guest kernel, crashing the guest this way doesn't win you anything you didn't already have.
† And now, post-fix, there's deliberately PROT_NONE guard pages around regions
...
> Because of the nature of system call filtering via seccomp, io_uring still presents a major security disruption in sandboxing.
This is pretty interesting as io_uring has been seen a lot of press as the hot new thing.
The basic gist seems to be that it's just not currently the upstream developer's focus
I think this article is a very good demonstration of why Rust is not a silver bullet. It was created with userspace applications in mind and a system application is an entirely different beats.
Think about it this way: in C it is easy to shoot yourself in the foot. But in kernel space you can easily blow up the entire building.
This bug exists because it's exactly C-style code manipulating buffers using raw pointers without length instead of slices.
Almost nobody thinks Rust is a silver bullet, but eliminating memory-unsafety is an absolute good; the more of it we can get rid of in the kernel, the better off we'll be.
Edit, I did learn cool new stuff tho, thanks.
The key here wasn't to produce an exploit. That would have been interesting, but ultimately not the entire goal. The key was to understand "how do we use Firecracker in the safest possible way for our use case?". To do that we picked one of the CVEs that looked like it could be exploitable and dug into it.
We learned a ton about Firecracker and KVM and walked away with some mitigations we can implement such that even if the bug had been exploitable the attacker would have more hurdles to jump through. Specifically, we'll be working to harden the guest operating system such that the untrusted code will have a difficult time escalating to root/kernel, which is a prerequisite for this sort of attack.
I walk through the process of developing the exploit and primitives, and was upfront that I ran into a mitigation which thwarted my exploit strategy. Similar to other exploit writeups I've done, I try to focus on the big picture and illustrate the idea (through writing and diagrams) while still being technically rigorous. Exploit development is much more reading code than it is writing it.
If you have any suggestions for improvement, or want to tell me which sections felt like handwaving to you, please let me know! Better yet, if you have an idea on how to defeat the mitigation so I can complete the exploit, I would love to discuss it.
BTW: Failing to produce an exploit for a very powerful bug like this, despite my best efforts, was considered a giant win for the security review of Firecracker.
>If you have any suggestions for improvement
Not GP, but I struggled to follow some parts due to passive voice.[0] There were a lot of sentences that omit the actor of the sentence, so I had a hard time understanding which component performed which action.
For example:
>If specified in flags, descriptors can be chained together with next containing the descriptor table index of the chained descriptor. virtio-vsock, buffers in a descriptor chain are used to construct a vsock packet. Something to note at this point: the buffer information in the descriptor comes from the guest, and it should be treated as untrusted.
So when I read that, I have to mentally walk back and figure out:
* can be chained together -> Who chains descriptors together?
* are used to construct a vsock packet -> Who constructs the packet?
* should be treated as untrusted -> Who should treat it as untrusted?
From context, I can figure these things out, but the Firecracker/kernel concepts you're explaining are occupying most of my mental bandwidth. Any bandwidth you can free up with simpler sentences makes it easier for me to focus on the main subject of the blog.
[0] https://writing.wisc.edu/handbook/style/ccs_activevoice/
I've read over this paragraph a few times now and I'm really struggling to see how their protections don't defend against the described issues -
"There are two problems that could occur; the base and result address may belong to two different regions, and the base address may not even exist in a valid region."
"the base and result address may belong to two different regions"
if addr >= region.guest_base && addr < region_end(region)
Surely region_end(region) stops it belonging to two different regions, as you're using one region in the for loop? I'm probably being thick!
"the base address may not even exist in a valid region."
Again surely if the addr < region_end(region) this would ensure it's within a valid region?
Is there any other info you can provide so that my simple brain can understand?
That's not accurate: While KVM is mandatory for Firecracker, it isn't for QEMU.
> Firecracker is comparable to QEMU
This is charitably true. They can be used for similar high-level tasks, and thus they are reasonably comparable.
> that utilize KVM
Both qemu and firecracker use KVM, it happens that qemu supports other hypervisors/options.
"utilize KVM" does not have to be read uncharitably as _only_ uses that.
It shouldn't be a problem for already knowledgeable readers, but it _might_ give false impression that QEMU only works as a KVM-only VMM monitor to others.