The Firecracker virtual machine monitor | Hacker News Reader

The Firecracker virtual machine monitor

lwn.net

96 pointsyankcrime16 comments

In addition to the firecracker-containerd[0] architecture, there's also firecracker support in kata containers[1]. While firecracker aims to be as lean as possible[2], kata aims to be portable[3]. It would be swell if the bits that run inside the VM and the protocol that goes over the vsock or vserial proxy were interoperable, but I see that as rather unlikely to happen soon.

0. https://github.com/firecracker-microvm/firecracker-container...

1. https://github.com/kata-containers/documentation/wiki/Initia...

2. https://firecracker-microvm.github.io/

3. https://github.com/kata-containers/documentation/blob/master...

The weird thing about Firecracker is that it doesn't run on Amazon EC2 shared host instances - but theoretically it will run on Digital Ocean shared instances, Azure shared instances and Google shared instances.

For the most part, Amazon EC2 is excluded from firecracker, unless you pay at the top end for bare metal EC2 instances.

Odd.

I believe this is only because ec2 does not allow nested virtualization. In my experience, nested virtualization is still buggy and can suffer from major performance issues. So although it is possible to run it in a vm, I'm not sure I would recommend running production code on it there.

Are you saying that Amazon does not support nested virtulization because it is too broken and is unusable technology. I've never heard that before.

Can you reference anything tangible to say that nested virtualization is too buggy to use in production?

Redhat seems wary. https://www.redhat.com/en/blog/inception-how-usable-are-nest...

"While Red Hat is now doing some level of QA for cascaded KVM, we are not supporting it - so it is clearly not meant for production use."

Which means absolutely nothing. Not for production use to them likely means they don't have a functional product to support it, not that it doesn't work.

Well given the expertise that RedHat has with KVM, it's a big warning "don't run nested kvm for now", it's pretty clear, what more do you need?

https://www.linux-kvm.org/page/Nested_Guests

"As of Feb 2018 this feature is considered working but experimental, and some limitations apply."

It is enabled by default in the latest stable kernel:

https://www.phoronix.com/scan.php?page=news_item&px=KVM-Linu...

My evidence is mostly anecdotal, unfortunately. Most of my experimentation was KVM in KVM from about 2012 and i saw frequent long pauses in the kernel, lockups, and kernel panics. At the time, the attitude from the qemu-kvm community was that it wasn't a high priority and that it may or may not work. It is possible that it is better now, but redhat's opinion[1] as of last year was still that it wasn't supported in production. I don't know if it is any better with other hypervisors. Last week, for example, I ran into an issue with nested virt in vmware fusion[2] that prevented me from running kvm in a guest. I would need a bunch of testing before I trusted it.

It looks like xen has spotty support[3] as well. Amazon uses a heavily forked version of xen, so I suspect support for it is even worse on their version.

(EDIT: added info on xen)

[1]: https://www.redhat.com/en/blog/inception-how-usable-are-nest...

[2]: https://bugs.launchpad.net/qemu/+bug/1661386

[3]: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xe...

Working on a hypervisor, I see this. It mostly crops up as things like IPIs being more costly when nested.

Nested KVM has been buggy since forever. Although a wonderful thing when it's working I'm not surprised Amazon doesn't allow it on their production systems. It has a nasty combination of: ability to crash the L0 hypervisor, different behaviour depending on features and family of the processor, and requiring very specialized skills to understand and debug.

Edit: If you do want to try testing nested KVM, then try my supernested project: http://git.annexia.org/?p=supernested.git;a=summary and Day 13 of the 2016 QEMU Advent Calendar https://www.qemu-advent-calendar.org/2016/

Edit 2: Don't confused nested KVM with nesting TCG. That's fine, albeit rather slow.

>> ability to crash the L0 hypervisor

Prove it and you'll get a bug bounty from Doogle and Microsoft won't you?

Why would Microsoft be interested? (no idea what "Doogle" is - do you mean Google? Why would they care about nested KVM?) Anyway when I find bugs in nested KVM I have reported them to the kernel developers or QEMU in the usual way and help any way I can to get them fixed.

Yeah, it is definitely because ec2 does not allow nested virtualization; however, all other major cloud providers do allow it.

I always assumed that the reason it was disabled had something to do with nested virtualization being less mature in xen-hvm than in kvm. I don't know if it is actually less mature in xen-hvm, but all of the other cloud providers are using other hypervisors so I figured xen-hvm was relevant. That said, Amazon has now released the m5, c5, etc instance types which run on kvm, but nested virtualization is still not supported.

Regardless of stability and performance, it's frustrating that they outright disable nested virtualization because it is useful in so many cases that aren't performance-critical where you really need to just run a VM: like building virtual machine images, adding another layer of isolation for untrusted software, or working with applications that cause kernel bugs (chromedriver a few years back was plaguing us with system deadlocks running at scale), etc.

I find it wonderfully ironic (and also just wonderful) that Amazon used some OSS written in Rust done as an experiment, and then forked it into production code running on AWS.

This also checks off a piece of software that I wanted to see written.

they used crosvm which is used on cromeos devices to support Crostini (linux vm)

Reply on news.ycombinator.com