Hacker News Reader

> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?

Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack:

* https://klarasystems.com/articles/virtualize-your-network-on...

You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host:

* https://man.freebsd.org/cgi/man.cgi?epair

But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

> Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?

Yes:

* https://wiki.freebsd.org/JailResourceLimits

> Can live processes be moved into or out of a jail?

No: jails are VM-like in functionality.

> Can jails be used to make a process think it's running as a different user?

The jail, being VM-like, would have its own passwd.

> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.

You can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs.

If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).

Well what are you going to do when you have to run something you don't trust?

Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.

Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.

Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.

If you really don't trust it, you run it inside a virtual machine.

The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.

A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.

Worth pointing out that VM escape is a thing too: https://en.wikipedia.org/wiki/Virtual_machine_escape

It is. It's a much smaller attack surface.

I'm unconvinced it is. All of those virtual hardware drivers require their own code, often at kernel privilege. Hypervisor functionality at the processor level helps but is a huge attack surface in itself. X86 "rings" are under tested because it's hard to - many violations just kill your whole VM, but more importantly there's a ton of instructions that access "physical" memory and the state of the page mapper, even at the ring-0 level, matters a lot.

I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.

It simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).

If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.

> I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /

When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?

For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.

I'm curious how that compares with what you're doing with jails.

No, my jailed deamons don't have an overlay, writes manifest to the filesystem the same as if they weren't jailed (which is good for me). Someone elsewhere described a vps they user which did have an overlay system, but I haven't used that.

Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like.

The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.

Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.

It's called VM-like because:

- Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1)

- Normally has its own network stack (VIMAGE)

- Normally runs a bunch of background services like a regular VM or on-metal would.

- Normally has the entire copy of user-land

- I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM.

The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux).

To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.

That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks.

The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.

I think you need to look into usage of "-like" suffix.

> That's just saying how people tend to use it

People tend to use words that describe how they use things to describe what they are.

Well, take a car as an example.

The question is not what you use a car for, but rather how the car is built.

Car is a carriage-like mode of transportation.

The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything."

I've provided examples why "VM-like" is used to describe jails.

Jails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.

> The defining attribute of a VM is that it runs its own kernel.

of a VM.

The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck.

Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place.

Or to put it another way, strcmp("VM", "VM-like") != 0.

If VM "like" means "doesn't do the one thing that VMs do aka run a separate kernel" it's not a very useful term

I believe I recall there being a period quite a while ago when things like LXC and OpenVZ were described/marketed as being "VM-like" around the same time that jails were popularized.

It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.

I don't know about any of that. LXC containers are also not VM-like.

> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.

> Jails are basically like a VM-like/light […]

With all due respect, they are not.

The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).

Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.

VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.

>> Can live processes be moved into or out of a jail?

>No: jails are VM-like in functionality.

Could this be implemented though if moving between sufficiently similar operating systems?

Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.

> * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?

Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation.

> * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?

I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this?

> * Can live processes be moved into or out of a jail?

A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails.

> * Can jails be used to make a process think it's running as a different user?

I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?

First of all - the Namespaces and Cgroups on Linux are not about security. They are about features.

FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed.

To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure.

Regards.

You're telling those namespace / cgroup features have nothing to do with security? Not in any way, shape, or form? I'm a little skeptical.

Namespaces and cgroups provide resource accounting and some limited isolation between trusted workloads. They're not considered a sandbox or security boundary because the processes have full access to the Linux kernel APIs, which are not well-hardened.

To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.

I agree containers are more for ease of deployment, not security. But security is about shades of gray… limited isolation is one of those.

Underrated post.

Hate to nitpick, but a comparison to bare namespaces wouldn't be apt, either, as Docker et al use more features than just namespaces in their container implementations, like control groups, union/overlay/snapshot file systems, nftables, etc.

For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.

> Can live processes be moved into or out of a jail?

That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa.

[1] https://www.dragonflybsd.org/features/#index9h2

The article is about how to set up jails on FreeBSD. The article has nothing to do with anything you are talking or asking about.