Comment by tptacek | Hacker News Reader

Comment by tptacek | Hacker News Reader

Regarding performance, I wrote a bit about this a few months ago in a post comparing different workload isolation techniques:

https://fly.io/blog/sandboxing-and-workload-isolation/

Two useful links from my Pinboard research for that post:

A person at Red Hat optimizing QEMU boot time:

http://oirase.annexia.org/tmp/paper.pdf

An Intel deck talking about qemu-lite:

http://events17.linuxfoundation.org/sites/events/files/slide...

(The other thing to follow up on if you're interested in the background on this stuff is kvmtool).

In both cases, a big part of the answer seems to be eliminating BIOS overhead; getting rid of oproms appears to have been the single biggest win for Intel. But the Red Hat article also finds lots of overhead in QEMU itself, and both pieces talk about kernel config issues (for instance, scrubbing the kernel you boot of subsystems that have expensive initcalls).

By comparison: Firecracker is purpose-built in Rust for this one task, provides no BIOS, and offers only network, block, keyboard, and serial device support --- with tiny drivers (the serial support is less than 300 lines of code).

Regarding your first link I had questions about the following passages from it:

>"People like to say “Docker isn’t a security boundary”, but that’s not so true anymore, though it once was."

Could you explain why that wasn't true and why it is now?

>"Systems security people spent almost a decade dunking on Docker because of all the gaps in this simplified container model. But nobody really runs containers like this anymore."

Could you elaborate? People don't run container like what? It isn't clear from the preceding paragraph.

Modern Docker running in default configurations is reasonably well locked down. I wouldn't use it for multitenant scenarios, though.

Can you elaborate why? Would love to know where it falls behind and how to fix these without going the Firecracker route.

With containers, both the kernel and the hypervisor are shared. With vms, only the hypervisor is shared.

It's a matter of having a smaller attack surface. There are plenty of container images that run with root access by default, which is almost full access to the kernel. This means that if the application running in the container is compromised, you need to rely on the kernel enforcing the sandbox between containers. This is a relatively new threat (root not being fully trusted), so beyond there simply being more attack surface, there's likely to be more bugs/vulns out there to be discovered. With effort and care you can safely run this but reducing attack surface is a good idea for defense in depth.

If one only allows container to run as a non-root user (no user namespace either) with all privileges dropped with strong mount isolation and some form of syscall filtering, then the attack surface is similar to that of hypervisors if not smaller while the performance is significantly better.

But yes, quite a few services assumes they have root privileges and do not work as is in such containers, like recent OpenSSH. For those cases VM isolation makes for much smaller attack surface.

The security models really aren't comparable. Again, see the blog post, which offers two examples of attacks that break the model you're proposing. I don't want to get into too much detail in this thread because I really just wanted to add some data to questions Julia Evans specifically asked in her post.

The blog post incorrectly states that Go is a memory-safe language. It is not in its standard configuration. And it does not mention that hypervisor bugs allowing to escape from a VM are typically due to wrong logic, not memory safety. Using memory-safe language does not protect from those. And that in turn is the reflection of complexity of hypervisor interfaces in modern CPU. And with VM one gets much greater exposure to hardware bugs as VM has access to more instructions.

Homer-into-the-hedges.gif.

The blog post linked upthread goes into some detail about why; answering that question is kind of the point of the post.

Not OP but most public cloud need stronger isolation than what Docker provides. GKE uses gVisor, AWS uses Firecracker now. When they started, lambda code was scheduled on a different EC2 instance per customer. VM isolation is just so much stronger. Firecracker hopes to make launching VMs as quick as launching containers while retaining the great isolation benefits of a VM.

> ...GKE uses gVisor

Another commentator on this thread points out that GKE uses trimmed-down Chromium OS? https://news.ycombinator.com/item?id=25884851

Edit: I guess container-optimized Chromium OS and gVisor are complementary (that is, you're welcome to run your app in gVisor sandboxed containers on a guest VM running Google's container-optimized OS). Ref: https://go-review.googlesource.com/c/playground/+/195983

See, for instance, https://cloud.google.com/blog/products/containers-kubernetes...

Thanks. I remember mrkurt say fly.io evaluated gVisor but settled on Firecracker eventually, though, I must ask: can gVisor on its own achieve same level of protection as micro-VMs? Even if not, can the protection offered by application kernels like gVisor and Kata be considered enough for multi-tenant workloads (like the one fly.io runs) inside the same guest (VM)?

Also, if I may, in what cases would one prefer to use Firecracker managed micro-VM running gVisor sandboxed containers? I'd imagine gVisor slows things down, so it might not be for everyone, but I'm curious as to what value-add might make gVisor worth that.

That's a tough question to answer! I spent some time on it in the blog post, and a lot more time that I didn't write about hacking on little bits of gVisor while researching that post. My take is, no, I don't think gVisor is as secure as Firecracker; it's way more ambitious and has a larger attack surface. But I buy that it is substantially better than OS-sandboxed container runtimes, and I would probably trust it for multitenant workloads.

(Kata, as I understand it, is mostly about microvm containers; I like Firecracker almost as much for its implementation, which I think is pretty gorgeous, as for the microvm design choice, but I like 'em both fine I guess. Also, there's a Firecracker-based Kata that people use, right?)

Do you know libkrun? github.com/containers/libkrun

"libkrun is a dynamic library that allows programs to easily acquire the ability to run processes in a partially isolated environment using KVM Virtualization."

Reply on news.ycombinator.com