Hacker News Reader

Comment by BossingAround | Hacker News Reader

Why not spin up containers instead of VMs? Seems to me containers would fit much better than VMs.

If the goal of the test is to debug a sad linux server, containers are going to severely limit what ways the server can be sad in, isn't it?

Can you give me an example of some of the severe limitations you're mentioning?

I don't think the DNS exercise would behave the same although that probably depends on how the container was setup. Docker usually controls /etc/resolv.conf. Another exercise is "try to figure out if you're in a container or VM so that'd definitely be different"

The question is not if the exercises would behave identically, but if you can test the objective in a container. For example, you can totally test, screw up, and fix DNS in a container. I would think that "try to figure out if you're in a container or VM" would be exactly the same as it is right now.

I can give you a bunch of things that can't be simulated in a container:

* Boot problems, such as: GRUB config/install errors, kernel parameters, init startup errors, blocking processes

* Many network scenarios, such as: PXE issues, multipath, load-balacing, anything requiring configuring network interface settings, firewall configuration.

* Resetting an unknown root password

* Booting directly to bash

* Filesystem mounts through fstab or systemd mounts

There's probably more I could think of, but I think that's a good list.

Containers have a history of escape vulnerabilities, for reasons like sharing a kernel with the host and other containers.

VMs are designed from the ground up to isolate guests, rather than focusing on application deployment.

Firecracker is the modern container alternative in untrusted compute scenarios, with Fly.io even converting container images into Firecracker VMs.

>Containers have a history of escape vulnerabilities

Generally agreed, but for this use-case do we care?

I haven’t gotten any of the challenges to load, but if you’re going to simulate a sysadmin it would make sense to give you high privileges (or even root) on the box. The more privileged you are inside a container, the more attack surface you expose.

Which is why you create a "dummy" host VM that hosts containers. Nobody's saying "host containers on your prod webserver." On the other hand, spinning up a VM for every user seems insane to me.

User mapping is now a standard feature in Kubernetes, so escape vulnerabilities aren't so much an issue anymore. Additionally, you can use gVisor.

User namespaces have resulted in multiple new container breakout CVEs in the last year. Some guides actually recommend disabling user namespaces because they are still somewhat new and perilous.

What are the ones in the last year? They provide security benefits as well. I mean, you could say the Linux kernel is also dangerous and the Windows kernel and pretty much anything that has ever had a CVE. You can also limit it to specific users too if that is a major concern.

You're talking about creating new user namespaces inside a container, not running a container in a user namespace. Running a container in a user namespace is strictly a security improvement over running it in the host user namespace.

Also, all container runtimes automatically block unshare(CLONE_NEWUSER) with seccomp already (unless they've disabled seccomp, which I'm not sure if Kubernetes still does).

Bypassing container security is easier than bypassing VM security.

Then wouldn't that be the ultimate test ;)

Reply on news.ycombinator.com