Hacker News Reader

Comment by jmclnx | Hacker News Reader

Very nice

I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.

View on HN

Jails are not, in fact, far more secure than Linux containers. Arguably they're a better, more coherent design, but both are shared-kernel isolation schemes, and the high-order bit of their security is exactly the same. The best argument you'll come up with is that FreeBSD kernel LPEs are rarer than those of Linux, but now we're just back to the standard Linux vs. FreeBSD security debate, which never goes anywhere.

I agree with all of this, with perhaps one tiny addendum. Defaults matter, and last I checked Docker/container defaults are miserable for security. I'm guessing if you randomly inspected production container vs jails out in the world, way more container solutions would have giant holes in them compared to jails.

Defaults on Docker specifically are pretty locked down. Even if if the container process has the root uid it is not able to do most things normal root can do, and is pretty much limited to being able to access root owned files and binding to privileged ports (which in a container doesn't tend to matter).

> Defaults on Docker specifically are pretty locked down.

That is strongly dependent on your threat model. The default docker configuration completely bypasses the firewall, making it trivial for containers to be exposed to the open internet with no way for admins to prevent it[0]. Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.

[0] It is quite easy for someone even slightly inexperienced to accidentally write, say `-p 1234:1234` instead of `-p 127.0.0.1:1234:1234` and thereby cause a security incident or near-miss; ask me how I know.

Docker has to run as root, or use otherwise insecure methods ("rootless" is a sham, it requires suid binaries and CVE ridden unprivileged user namespaces).

I agree with ports, working[0][1][2] on it.

[0] https://github.com/moby/moby/discussions/45524

[1] https://github.com/moby/moby/issues/45532

[2] https://github.com/moby/moby/pull/45076

That said, docker does have support for running "rootless"

> Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.

I never got why this is commonly used as an argument against Docker, TBH. You just don't give out access to the Docker socket to anything untrusted. Doesn't pretty much everyone know that by now?

I feel like people always say Docker is awfully insecure, but then the proofs-of-concept include flags like `--privileged`, or the socket is mounted, or / is mounted, or --net=host is set... etc. Docker by default always seemed pretty good to me, but I'm not very experienced in that realm, so I'm just wondering what I'm missing.

Please provide technical details of `Jails are far more secure`.

Multiple CVEs for escaping? E.g.:

* https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe...

* https://securitylabs.datadoghq.com/articles/dirty-pipe-conta...

* https://snyk.io/learn/docker-security/top-5-vulnerabilities/

* https://www.container-security.site/attackers/container_brea...

The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs:

* https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...

No? Because many of them Docker specific. Neither I mentioned Docker anywhere, not Docker is a good example of software with a good security record. Another link lumps together k8s, runc and so on. I specifically mentioned systemd-nspaw in the beginning as contender to jails, not other solutions.

Misleading: the most important vulnerabilities that break shared-kernel isolation systems like Jails, Zones, and containers are kernel LPEs, and they aren’t reported as “escapes”.

What about `firejail` on Linux? Is it not much more secure than FreeBSD Jails which is just based on chroot?

FreeBSD Jails are conceptually just based on chroot. Their actual design and implementation are a few orders of magnitude more secure.

A few order of magnitudes? Can you be more specific? I'm genuinely wondering what you could be referring to that I've missed.

chroot more or less isn't secure at all. All they're really saying is that Jails aren't built on top of chroot, but rather are a ground-up reimplementation motivated by improving chroot.

Ah! For some reason I thought jails were built on something similar to chroot at the kernel level, meaning that it also came with the same security caveats.

(to be more accurate, I knew that it was not "on top" in the same way as Linux containers are built on top of chroot. )

https://en.wikipedia.org/wiki/Operating_system-level_virtual...

https://en.wikipedia.org/wiki/Freebsd_jail

Also the SANE paper:

http://therbelot.free.fr/Install_FreeBSD/jail/jail.pdf

Thanks, I'll check out the specifics.

What makes Jails so far ahead of containers in your opinion?

Not op, but for me:

You can setup Linux in a jail.

You can nest bhyve virtual machines in a jail.

You can assign individual NICs to a jail.

You can encrypt with ZFS a jail.

You can run browsers in a jail with full set of features.

All of those available on Linux and even more, see https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.h...

I don't see the guide to run FreeBSD in a container from that man page.

And what about if my Linux Distro doesn't use system-md, like Slackware.

> And what about if my Linux Distro doesn't use system-md, like Slackware.

It's a choice of Slackware not use systemd infra. However on Linuxes you have more that one choice to run containers, your next option would be LXC/LDX https://docs.slackware.com/howtos:misc:lxc

Sure and my choice not to use it too. Well aware of LXC too.

I'm not claiming Linux can't. I was stating why. I know Linux can do what FreeBSD can do, but BSD does it better.

However both are pretty much mimicked concept from Solaris Zones

> However both are pretty much mimicked concept from Solaris Zones

Your timeline is off.

FreeBSD and Linux were within a year of each other (not in-tree on linux, but Debian packaged VServer kernels and VServer + GRSec kernels; we used vserver+grsec debian packaged kernels at work in the mid through late '00s).

Solaris containers came years later.

2000 - FreeBSD jails

2001 - Linux VServer

2004 - Solaris Containers

https://blog.aquasec.com/a-brief-history-of-containers-from-...

I wonder why would anyone spend time on developing and maintaining that. FreeBSD needs Linux userland and syscall support to get roughly the same level of software compatibility. The opposite isn't true (maybe it is for your special snowlake internal application developed specifically for FreeBSD, but it isn't so in general case).

All other things you have mentioned are possible with pretty much any of the Linux container runtimes. systemd-nspawn is just one of them, and I don't think it is actually used very much compared to the alternatives.

Most of those are supported by "containers", which are after all just a chroot and a process tree with restricted visibility into certain kernel objects. Assigning individual NICs or running a browser with extra isolation is one of the classic Linux namespace tutorials.

  > You can setup Linux in a jail.

I don't see how this can be possible. Could you explain more how to boot a Linux kernel within a FreeBSD jail?

edit: I'm not talking about running binaries compiled for Linux under FreeBSD. The parent said it's possible to set up Linux within a jail, so I want to see instructions to boot an actual Linux kernel as a FreeBSD process.

You use FreeBSD kernel and Linux userland. FreeBSD can emulate Linux system calls.

There is no such thing as “Linux userland”. Obviously you can port most GNU utilities and other things that are typically run on Linux on FreeBSD but that’s a far cry from “running Linux”.

Using the linuxulator.

https://wiki.freebsd.org/LinuxJails

> Assigning individual NICs or running a browser with extra isolation is one of the classic Linux namespace tutorials.

With FreeBSD you can set up a standalone virtualized network stack:

* https://klarasystems.com/articles/virtualize-your-network-on...

* https://man.freebsd.org/cgi/man.cgi?vnet

* https://wiki.freebsd.org/Jails/VNET

I read the quote you're replying to as "you can also do this under linux", i.e. it didn't dispute that freebsd jails can do it. But it's not a distinguishing feature of jails.

You could maybe build and run User-Mode Linux on FreeBSD, and run that in a jail. That's pretty close to your own interpretation of "Linux kernel in a jail"

But to be real: that's obviously not what was meant. If someone wants to install and run Ubuntu inside a FreeBSD jail, it is well possible to do that. Nobody really cares if it's running kernel.org code, just that the binaries are running as expected.

I don’t know about the low level stuff but LXC is pretty much the same end result as FreeBSD jails. At least for my very basic needs that is.

Reply on news.ycombinator.com