Kata Containers: Virtual Machines that feel and perform like containers

Would it be simple enough to stick a union file system over the top of the host file system?

Yeah, it's worth understanding the attack surface of DAX - if someone has information I'd be very interested. That said, you could mitigate it in other ways depending on your use case.

Having gone through an evaluation of Firecracker's security my main conclusion was that sandboxing the processes in the guest is the highest 'bang for your buck' way to reduce escapes.

I love the vagrant UX, I’d be thrilled to see a refresh for 2023. Box management is odd. Weaveworks ignite/footloose are also good but don’t quite scratch my itch. I think vagrant UX with ignite style kernel/rootfs as oci image is a nice flow.

I also use cloud init and one can quickly (5 seconds) spin up VMs with it. I use virt-install which has command line options for the cloud init related files which makes everything very easy.

On the onboarding point: Almost all projects would do themselves a big service by putting work into the onboarding experience because it's a key factor in how many people will use and help developing the project.

> Last time I looked (a few months ago), the documentation was pretty sparse or outdated.

It still is, though it works somewhat seamlessly when installing with https://github.com/kata-containers/kata-containers/blob/main...

Though only one of the hypervisors works well.

Yeah I'd like to see some numbers on that, like startup time.

I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight to start (especially when run via Kubernetes).

The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. This can practically limit the number of Katacontainers you can run to something much smaller than is possible with ordinary containers, since RAM is the constrained resource on most servers.

Nevertheless, with confidential computing coming along, it's likely that at some point in the future many containers will really be VMs, since current CPUs implement confidential computing on top of existing VM primitives (and that's basically necessary due to the way the guest RAM is encrypted). It's likely that any workload that touches PII, finance, health, etc will be required to use confidential computing.

there is pretty low overhead if you are opinionated - this is very similar to firecracker (AWS) tooling, so cut down hypervisor with ~ 0 devices, and a cut down guest OS means pretty quick boot times

there's probably a big asterisk there. the correct term is probably "fast enough"

virtualization adds very overhead, a Windows VM running with a dedicated GPU can get 95% of the host's score on 3dmark.

the biggest issue on these cases is IO which can be handled in a few ways.

The short version is that kernels support guest/host relationships natively so that guests can pass operations directly to the host without having to go through an additional system call. Everywhere you do this is attack surface where an attacker in the guest can communicate with privileged facilities, so you want to minimize this where you can.

There's usually overhead in the places where the communication requires an additional hop. If you want your host filesystem isolated you're going to need a translation layer and it will be slower. If you're willing to open up your host OS's filesystem, you can basically get ~0 overhead.

Quantum hack?

Nah, it’s 2023, brute it

You mean like http://www.lambdashell.com/ ?

> I suspect that nearly all containerization software is insecure.

This seems, well, naive?

You think there aren't a lot of people that have tried to break out of cloud containers?

Both EC2 and gcloud have had issues over the years with container breakout and leaks.

Complex (ie literally layers of operating systems) software has bugs. Yes. But so does the non-containerised base-case.

We have the best style of honeypots you could ever ask for already running- payment infrastructure on the internet. Go get 'em.

That honeypot was called VPS hosting and over ten years it didn't get hacked much if at all.

Lots of people have tested these systems.

Here's work we did to exploit Firecracker. We had a known, promising vulnerability, and still failed to break out.

https://web.archive.org/web/20220927150915/https://www.grapl...

Exploitation is extremely hard. No one has ever exploited RowHammer in the wild, to my knowledge, and there are a lot of reasons why - but even still, RowHammer isn't magic, you can mitigate against it by increasing your refresh rate, or by limiting the attacker's execution time. Not to mention that vendors have deployed numerous methods of reducing the likelihood of an attack and it is quite complex to pull off these days.

Those are opensource, you can run them yourself, no need for a "honeypot" hosted by someone else.

And I have no idea why you think no one is trying to break them or audit them.

Apologies everyone, I've been a little too close to my work lately and jumping to conclusions. A normal person might have phrased this like:

Hey that's an interesting solution you have there, do you think that emulation might help guard against side-channel attacks? Do you have any plans to mitigate them? I wonder if anyone has gamified container security, maybe with a honeypot somewhere where people could try to break out.

If a game anti-cheat system can detect that a regular VM is a VM, then it will also detect Kata VM is a VM. In both cases you're running a game in a VM with a dedicated kernel.

Kata VMs are especially VM-y because they use a lot of VM-only features that wouldn't work with real hardware to enhance performance by sharing work between the guest and the host.

Most VM detection I've seen is done through fingerprinting. Checking what the hardware is, certain CPU features/flags, config files, etc.

Not in EKS. EC2 doesn't support nested virtualization today.

Containers are not appropriate for some scenarios. If your threat model is "attacker has arbitrary code execution" you should be very wary of a container holding them for long - you're one kernel privesc away from a container escape. It's nice to have if your situation is "first the attacker would need to exploit my service and then they'd have code exec", but if you provide RCE as a service a normal container is just not a good barrier.

By contrast, running attacker code in Firecracker is much safer - for one thing, the attacker needs to escalate within the VM in order to then expose the necessary primitives to then escape the VM. So it immediately adds an additional required vuln. But also, instead of the entire Linux kernel being the attack surface (it is for the first vuln, but...) you have to attack a much smaller codebase that implements the VM.

In the case of Firecracker this is really hard and, depending on your exploit, if you end up controlling the Firecracker VM itself you're actually in yet another sandbox - so you now need another vuln to escape (although that sandbox is not amazing imo).

Many serverless services use MicroVM rather because they think containers doesn't have enough isolation (=insecure?) to run untrusted code.

Run multiple workloads on the same CPU that you don't fully trust. VM-based containers will force the CPU to properly context switch between guests (e.g. using VMENTER on x86), normal process-based containers won't.

That's what this is. "Containers" implemented as actual VMs.

The API for containers is widely used.

1. So you can run any container image that someone made into a Docker container instead of building your own VM images

2. So you can run in Kubernetes and use the Kubernetes API, alongside any non-VM Pods and many other features in your Kubernetes cluster

3. Because traditional hypervisors have a higher overhead cost than the microVMs used by Kata and Firecracker

4. Because traditional hypervisors have a longer startup time (e.g. Firecracker is used by Amazon to run Lambda functions)

They are VMs, but without the annoying "pet" UX.

If you install kata with https://github.com/kata-containers/kata-containers/blob/main...

Then you just use it with:

docker run --runtime io.containerd.kata.v2 --rm -it hello-world