Ubuntu Linux Jail on FreeBSD 12.2-Release | Hacker News Reader

Ubuntu Linux Jail on FreeBSD 12.2-Release

wiki.freebsd.org

205 pointstodsacerdoti82 comments

I'm hoping this will eventually allow running Linux video conferencing software, such as Zoom or Teams. I tried running Zoom the other day in this sort of jail but kept running into kernel panics.

Nice progress, but it notes that `systemd` doesn't work, which means all the things that depend on `systemd` won't work either.

Systemd is emphatically Linux-specific software and relies on almost every single non-POSIX syscall under the sun. FreeBSD's Linux emulator isn't complete enough to run systemd at this time (chiefly missing things like cgroups and namespaces). It is theoretically possible we could eventually emulate enough of Linux to run systemd.

As far as things that depend on systemd — sort of. Sure, you won't be able to manage servers with systemctl and that sort of thing; systemd as pid 1. Smaller pieces under the systemd umbrella, of which there are several, may work well enough.

I've been generally OK with systemd, and think most of the complaints about it were over-dramatic, but this is exactly one of the things we were warned about, right?

systemd-as-SysV-init replacement was fine. It's all the other stuff that many people think is overkill: udevd being pulled in, journald, time sync, file system mounting, network management, etc.

And all are in one repo with tight coupling.

For an "init system".

systemd is like that non-lazy guy we were warned about:

I distinguish four types. There are clever, hardworking, stupid, and lazy officers. Usually two characteristics are combined. Some are clever and hardworking; their place is the General Staff. The next ones are stupid and lazy; they make up 90 percent of every army and are suited to routine duties. Anyone who is both clever and lazy is qualified for the highest leadership duties, because he possesses the mental clarity and strength of nerve necessary for difficult decisions. One must beware of anyone who is both stupid and hardworking; he must not be entrusted with any responsibility because he will always only cause damage.

https://en.wikipedia.org/wiki/Kurt_von_Hammerstein-Equord

But systemd was never just an init system. AFAIK, systemd has always been advertised as a system and service manager for Linux. With that in mind it makes perfect sense that systemd manages what it does.

Whether or not such an architecture is desirable is debatable, but the portrayal of systemd as just an init system is a fundamental misunderstanding of what systemd is.

> With that in mind it makes perfect sense that systemd manages what it does.

It made perfect sense to pull in udevd, which had existed externally for a long time, and tightly couple it so that it can no longer be used independently? (The Gentoo folks forked eudevd.)

Even if you stripped systemd down to the "as-SysV-init replacement" part it still wouldn't run on FreeBSD, because systemd uses cgroups for that.

People keep saying it's tightly coupled, but never supply any proof for that. You can still run udev without systemd, the NTP stuff is completely optional and in separate binaries and the same is true for the network management stuff. The journal is indeed required, but any traditional syslog daemon still works, and you can trivially disable the persistent journal (I even believe that's the default on Debian).

Suppose I want to write my own competitor to systemd-resolvd. I want it to work with systemd-networkd, in the same way as systemd-resolvd does. Or just patch dnsmasq or unbound to be able to do that. Is there a stable interface for that? An RFC I can read? Any guarantee they won't change the interface and break my version, while they update both of theirs together?

> Any guarantee they won't change the interface and break my version, while they update both of theirs together?

No, I think you can guarantee that the interface will change at some point.

Judging from the manpage and a quick look at the code, the only thing for which systemd-networkd and systemd-resolved interact is to get the DNS servers pushed over DHCP, which it does over the D-Bus interface that falls under the interface stability promise[1].

[1] https://systemd.io/PORTABILITY_AND_STABILITY/

That is not the only thing for which they interact, see e.g. systemd.network(5). The systemd-resolved.service itself reads the systemd.network config files. This is what people mean by tightly coupled. But more importantly, you're telling me what it does currently, not what it's going to do tomorrow.

Suppose it did currently only interact through a stable D-Bus interface. What stops them from adding some new interaction and supplementing or replacing the existing one with it? You're left with a guarantee that systemd-resolved will continue to support the old API that systemd-networkd may stop using.

Please checkout the progression of Devuan if you are wanting to understand just how viral systemd is in modern linux. If it wasn't so tightly coupled, why did it take 2 years to decouple it?

https://sysdfree.wordpress.com/2017/08/16/112/

https://nosystemd.org/

First of all, Devuan wants to get rid of any and all systemd code. This includes for example libsystemd-daemon0, which is a tiny library that implements the stable notification protocol and is a no-op on systems not running systemd. You can reimplement it yourself in pretty much any language, but lots of programs just link against the systemd library because it's easier and has no tangible downside, even for people not running systemd. Eliminating such dependencies isn't particulary hard but takes a lot of work. Likewise, the rest of the world has left SysV init scripts behind, so they have to write those too.

The second reason is that systemd provides features for which there simply is no alternative. Desktop environments are entangled with logind, because it's the only software providing that functionality. Before logind they had a hard dependency on ConsoleKit, which was pretty much abandoned. Reimplementing all these features takes time.

If you are referring to the lack tight coupling being the ability to remove journald, time sync, etc, then you are correct. I should have been more clear I was talking about systemd as a whole, and not just those other subsystems, but I'll assume you meant the latter, in which case, yeah, it's possible to stop using lots of the systemd subsystems without too much pain (which I have done quite frequently, and found great stability improvements.) To be frank, I just really don't like the handwaiving away of valid criticisms people have (on any topic), with such dismissive language as "People keep saying ... but never supply any proof for that." This to me reeks of intellectual dishonesty, but again, that was my response because I probably misunderstood you to say that about systemd as a whole instead of how you meant it which was focused on gp's comment about subsystems.

One note: many people pretend sysv init has been dead and left behind, but I've seen production systems in major places still using it (not just embedded either). It's not nearly as dead as people might like to think.

> you can trivially disable the persistent journal (I even believe that's the default on Debian)

The persistent journal is enabled by default in Debian testing.

It sounds like systemd wants to be an OS kernel.

No, it wants to be 90% of the OS except the kernel (the other 10% is libc and a package manager).

systemd provides process and resource isolation. You can either have something comparable with everything or something the uses specific features of the kernel. You just can't have it all.

Not much of an issue - the world goes on without systemd.

You'll meet a few annoying dependencies on it, but as one will quickly find, it's rarely critical, and mostly because of people using its dbus implementation or similar (which has no relation to the rest of systemd, and yet is entangled in libsystemd...).

The main purpose of Linuxulator is to run software that's not available natively, which basically means closed-source software, eg. Steam, or native Chrome, or Eagle. Pretty much none of it depends on systemd.

What you would need systemd for is virtual machines - and that case is better handled by just running Linux under bhyve(8).

This wiki page does not tell you how to setup a jail for the linux OS, so the name is bogus.

Yeah, I'm not sure what the exec.start command would be for systemd-based Ubuntu.

Depends on what you want from the jail. If you want to be able to ssh into it, then make it start ssh ("service ssh start") and that's it.

In order for the linux compat to be generally useful, we really need to get electron apps working, which means getting chrome itself working.

There are some commercial software packages that run which are useful in more limited scopes. For example, I run a linux PVR server (SageTV), and Intel's Vtune GUI. But getting chrome & electron working would enable tons of commercial linux apps.

Running Linux Chrome in a FreeBSD Linux jail already works [1].

But, Chrome is already ported to FreeBSD [2]. Not sure why you'd need Linux Chrome specifically. In fact, there's already a port of Electron [3].

[1] https://forums.freebsd.org/threads/linuxulator-how-to-run-go...

[2] https://www.freshports.org/www/chromium

[3] https://www.freshports.org/devel/electron5/

Ah, I think the key part from [1] is: "--no-sandbox --no-zygote", and possibly "--test-type". Cool, thanks.

It does not look good for the Electron port [1].

These ports will be removed soon due to their Python 2 dependency, and the available versions are not supported anymore [2].

[1] https://www.freshports.org/search.php?query=electron&search=...

[2] https://www.electronjs.org/docs/tutorial/support

Its marked as broken in the LinuxApps page linked to from the article (https://wiki.freebsd.org/LinuxApps). And its been broken every time I've tried it on -current.

You need Linux electron working in order to be able to run things like Slack's Linux desktop app, and other electron apps which are compiled for Linux.

And you need the Linux binary chrome from Google in order for DRM to work.

Worth to make note of the fact that the forum post that was linked to by the parent comment to yours, about being able to run Linux Chrome on FreeBSD, was actually posted today. So it might be promising? But at the same time, the post also states that it relies on a few workarounds that are likely to interfere with other things on the system. So probably one should not get their hopes up too high just yet either.

Have you tried it with --no-sandbox --no-zygote --test-type yet? I haven't, yet. Admittedly, I'd much rather run Chrome in its sandbox, but if this is a good starting point, great.

You can NOT use Netflix/Spotify/... on Chrome on FreeBSD.

You CAN use Netflix/Spotify/... on Chrome under Ubuntu Linux Jail on FreeBSD.

Reason: Widevine DRM

BSD subsystem for Linux.

Pretty much. Long predates WSL1, though, and facing many of the same difficulties that prompted the Windows team to just run Linux in a hypervisor (WSL2).

> facing many of the same difficulties that prompted the Windows team to just run Linux in a hypervisor

How's that?

WSL had serious performance issues due to design differences between linux and NT. Linux and freebsd both being unices, there is no such impedance. Some features (like cgroups) remain unimplemented, true, but there haven't been any big difficulties afaik.

Also interested in the response to the parent (loeg)

> facing many of the same difficulties that prompted the Windows team to just run Linux in a hypervisor

If by difficulties, you mean implementing system calls, this is being kept tabs on here: https://wiki.freebsd.org/Linuxulator

One that comes to mind is FreeBSD's kqueue, which opens a file descriptor for every file watched [1]

So inotify, inotify_add_watch, inotify_rm_watch isn't started. Lots of applications need watching of changes on directories. Even plain old FreeBSD gets messed since a large webpack project is going to chew up way too much.

If you're interested in this type of system call across operating systems check out https://github.com/emcrisostomo/fswatch

On WSL1/2:

Namely in WSL1, there were serious issues with package systems like npm, node_modules/ would have file descriptors get clogged, and it took a full system reboot to get WSL working again: https://github.com/microsoft/WSL/issues/1529. WSL2 Fixes it.

[1] https://emcrisostomo.github.io/fswatch/doc/1.8.0/fswatch.htm... See Freebsd -> kqueue -> Peculiarities

Sure, "BSL" doesn't have all the same difficulties as Windows WSL1 had. There are still design mismatches between FreeBSD and Linux that make implementing Linux features difficult (just for example, compare Linux clone() to FreeBSD fork()), and there are many, many features that remain unimplemented (inotify, cgroups, namespaces, ...).

I agree it's largely "just" a matter of implementing the missing functionality, which has a large surface area and requires building novel infrastructure in the FreeBSD kernel.

I'd say it doesn't have ~90% of the difficulties WSL1 had. From what I remember WSL1 had problems with even most basic functionality, like the filesystem semantics, and never got to support eg DRI for X11.

Regarding clone() - it's rfork(2) you want to compare to, not fork(2). Not the same, sure, but much closer.

(Disclaimer: I'm one of the people working on it, so I'm obviously biased.)

Not just performance issues. WSL1 had so many limitations that it was silly.

WSL2 gives you pretty much the whole linux environment. A truly future of Linux of desktop...

Linux subsytem for FreeBSD you mean?

That would make more grammatical sense, but the parent comment is presumably referencing Windows Subsystem for Linux[1], which is a subsystem for Windows that runs Linux.

1. https://docs.microsoft.com/en-us/windows/wsl/about

I wish there was a usable Docker Engine API implementation that could run containers in jails using zfs for image layering. There is a Docker for FreeBSD port, but last time I checked, it was super old. Having Docker as a first class citizen on FreeBSD would make it really incredibly as a server OS.

Having played with SmartOS containers using linux compatibility (not docker). I will say this: Use native containers for that OS. The slight differences that you will find, will drive you up the wall. They will be esoteric and happen at the worst time.

And i should say I love SmartOS, it is a really cool bit of tech... I just love simplicity more, and a compatibility layer is never simple.

https://en.wikipedia.org/wiki/SmartOS

If linux comes out with tools remotely approaching the quality of either freebsd or solaris planning everyone should drop what they're doing to go watch. Linux ain't gonna improve itself, and the pains of docker are an excellent entry to pressuring linux devs to do anything that can improve userspace experience, an otherwise deeply unlikely prospect. It took them more than a decade to implement basic jailing after BSDs and the tech is still, shall we say, extremely rough....

Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO.

The ancient rhel something something doesn’t like the debian other thing, and I even saw a changelog about broken kernel compatibility and flipping a flag in proc or cmdline but then I forgot what it said.

Whatever. Life is short. Docker is in no way independent of the host kernel. Spend time debugging something else.

SmartOS is great though. I wish they’d take over the world.

To give this argument some more meat with an example: some older glibc versions use vsyscalls and not vDSO. And kernels in some modern distributions disable vsyscall emulation. This is a problem because people use CentOS 5 or 6 images to build binaries that are compatible with many historical glibc versions.

If the host system does not enable vsyscall emulation, running binaries on such old distributions will lead to segfaults. (Which can be resolved by booting the host kernel with vsyscall=emulate.)

> Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO.

I would love any details you can provide; that sounds interesting. (I'm an OS nerd)

Maybe not docker, but there is pot: https://potluck.honeyguide.net

Which aims to provide something similar as Docker for FreeBSD.

I'm the one behind pot, a framework to manage jails as containers. Here, there is the presentation on how to orchestrate jail using nomad, from Hashicorp: https://archive.fosdem.org/2020/schedule/event/orchestrating...

pot is nothing more than a Proof of Concept, a model that uses FreeBSD technologies to implements a typical container workflow: jail, ZFS, pf, rctl and cpuset.

There is no compatibility to OCI standards, but I believe it would be possible to implement them.

There's also Bastille, which looks very interesting: https://bastillebsd.org/

Every couple of years people express interest but no one has ever done the work. It is very much doable as the original port showed, and the zfs driver already exists for image layering as that works on Linux.

I think that the people who work on it have goal to run linux in it and that causes the project to be more complex than it should. For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there, so why it should be different with FreeBSD? Yes, it would be great to run the same docker images on FreeBSD, but that should be a later step, and stuff like described in this wiki, could be used to accomplish that.

For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there,

Actually, it would with qemu user-mode emulation. This can be done transparently with binfmt_misc, which is a one or two-liner on many distributions.

BuildKit now can now also automatically without any additional setup:

https://github.com/moby/buildkit/pull/1516

But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images.

> But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images.

This is really an important point, MS failed to get WSL1 good enough so WSL2 was just a plain Linux kernel being virtualized. Maybe running a some partially virtualized linux kernel(Linux already has modes for this?) via bhyve relying on a host FreeBSD filesystem (using the already developed layering) might be an feasible approach that requires less Linux API chasing while retaining the "native" FS advantage at the expense of some extra memory (bhyve) ?

From what I understand, with qemu-usermode you're still risking incompatibilities, as QEmu needs to be able to handle the syscalls to translate (or 'transmarshal', if that's a word) each of the syscalls.

There has been some work done: https://reviews.freebsd.org/D21570. It appears stalled, though.

There used to be a thing like this called jetpack [1], but unfortunately the project has died.

[1] - https://github.com/3ofcoins/jetpack

Maybe I'm wrong, but isn't this a key component required to have native support (without having to use a VM?)

Frankly, though docker failed to deliver most of its promises, but at least it became a standardized deployable unit.

macOS seems to do alright

MacOS runs a VM with Linux installation on it. That's why it takes so long to start, it is slow and uses so much CPU power and memory to run.

Not to mention the unusable volume performance

Docker on macOS uses a VM for the kernel. Its much slower than on a Linux host

About 6 months ago I tried running FreeBSD on a network appliance, when I usually run Debian. Wow! I'm never going back. Everything about it made me really happy, from the smooth install (Debian often has issues with serial console installs), to being able to boot ZFS, to the sheer simplicity of configuring the OS.

One stark example of how clean the FreeBSD install is: there are something like 9 or 10 processes running after boot for a fresh FreeBSD install, compared to dozens for a fresh Debian or Ubuntu install.

Using jails is also a lot more pleasant than using docker.

True, there is a point that pfsense and OPNsense are the best free Firewall Appliaces.

Reply on news.ycombinator.com