Hacker News Reader

> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.

You can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs.

If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).

Well what are you going to do when you have to run something you don't trust?

Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.

Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.

Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.

If you really don't trust it, you run it inside a virtual machine.

The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.

A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.

Worth pointing out that VM escape is a thing too: https://en.wikipedia.org/wiki/Virtual_machine_escape

It is. It's a much smaller attack surface.

I'm unconvinced it is. All of those virtual hardware drivers require their own code, often at kernel privilege. Hypervisor functionality at the processor level helps but is a huge attack surface in itself. X86 "rings" are under tested because it's hard to - many violations just kill your whole VM, but more importantly there's a ton of instructions that access "physical" memory and the state of the page mapper, even at the ring-0 level, matters a lot.

I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.

It simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).

If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.

> I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /

When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?

For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.

I'm curious how that compares with what you're doing with jails.

No, my jailed deamons don't have an overlay, writes manifest to the filesystem the same as if they weren't jailed (which is good for me). Someone elsewhere described a vps they user which did have an overlay system, but I haven't used that.

Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like.

The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.

Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.

It's called VM-like because:

- Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1)

- Normally has its own network stack (VIMAGE)

- Normally runs a bunch of background services like a regular VM or on-metal would.

- Normally has the entire copy of user-land

- I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM.

The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux).

To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.

That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks.

The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.

> That's just saying how people tend to use it

People tend to use words that describe how they use things to describe what they are.

Well, take a car as an example.

The question is not what you use a car for, but rather how the car is built.

Car is a carriage-like mode of transportation.

The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything."

I've provided examples why "VM-like" is used to describe jails.

I think you need to look into usage of "-like" suffix.

Jails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.

I believe I recall there being a period quite a while ago when things like LXC and OpenVZ were described/marketed as being "VM-like" around the same time that jails were popularized.

It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.

I don't know about any of that. LXC containers are also not VM-like.

> The defining attribute of a VM is that it runs its own kernel.

of a VM.

The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck.

Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place.

Or to put it another way, strcmp("VM", "VM-like") != 0.

If VM "like" means "doesn't do the one thing that VMs do aka run a separate kernel" it's not a very useful term

> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.

> Jails are basically like a VM-like/light […]

With all due respect, they are not.

The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).

Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.

VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.

>> Can live processes be moved into or out of a jail?

>No: jails are VM-like in functionality.

Could this be implemented though if moving between sufficiently similar operating systems?

Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.