Hacker News Reader

Comment by djsumdog | Hacker News Reader

I wish there was a usable Docker Engine API implementation that could run containers in jails using zfs for image layering. There is a Docker for FreeBSD port, but last time I checked, it was super old. Having Docker as a first class citizen on FreeBSD would make it really incredibly as a server OS.

View on HN

Having played with SmartOS containers using linux compatibility (not docker). I will say this: Use native containers for that OS. The slight differences that you will find, will drive you up the wall. They will be esoteric and happen at the worst time.

And i should say I love SmartOS, it is a really cool bit of tech... I just love simplicity more, and a compatibility layer is never simple.

Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO.

The ancient rhel something something doesn’t like the debian other thing, and I even saw a changelog about broken kernel compatibility and flipping a flag in proc or cmdline but then I forgot what it said.

Whatever. Life is short. Docker is in no way independent of the host kernel. Spend time debugging something else.

SmartOS is great though. I wish they’d take over the world.

> Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO.

I would love any details you can provide; that sounds interesting. (I'm an OS nerd)

To give this argument some more meat with an example: some older glibc versions use vsyscalls and not vDSO. And kernels in some modern distributions disable vsyscall emulation. This is a problem because people use CentOS 5 or 6 images to build binaries that are compatible with many historical glibc versions.

If the host system does not enable vsyscall emulation, running binaries on such old distributions will lead to segfaults. (Which can be resolved by booting the host kernel with vsyscall=emulate.)

https://en.wikipedia.org/wiki/SmartOS

If linux comes out with tools remotely approaching the quality of either freebsd or solaris planning everyone should drop what they're doing to go watch. Linux ain't gonna improve itself, and the pains of docker are an excellent entry to pressuring linux devs to do anything that can improve userspace experience, an otherwise deeply unlikely prospect. It took them more than a decade to implement basic jailing after BSDs and the tech is still, shall we say, extremely rough....

Maybe not docker, but there is pot: https://potluck.honeyguide.net

Which aims to provide something similar as Docker for FreeBSD.

I'm the one behind pot, a framework to manage jails as containers. Here, there is the presentation on how to orchestrate jail using nomad, from Hashicorp: https://archive.fosdem.org/2020/schedule/event/orchestrating...

pot is nothing more than a Proof of Concept, a model that uses FreeBSD technologies to implements a typical container workflow: jail, ZFS, pf, rctl and cpuset.

There is no compatibility to OCI standards, but I believe it would be possible to implement them.

There's also Bastille, which looks very interesting: https://bastillebsd.org/

Every couple of years people express interest but no one has ever done the work. It is very much doable as the original port showed, and the zfs driver already exists for image layering as that works on Linux.

I think that the people who work on it have goal to run linux in it and that causes the project to be more complex than it should. For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there, so why it should be different with FreeBSD? Yes, it would be great to run the same docker images on FreeBSD, but that should be a later step, and stuff like described in this wiki, could be used to accomplish that.

For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there,

Actually, it would with qemu user-mode emulation. This can be done transparently with binfmt_misc, which is a one or two-liner on many distributions.

BuildKit now can now also automatically without any additional setup:

https://github.com/moby/buildkit/pull/1516

But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images.

> But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images.

This is really an important point, MS failed to get WSL1 good enough so WSL2 was just a plain Linux kernel being virtualized. Maybe running a some partially virtualized linux kernel(Linux already has modes for this?) via bhyve relying on a host FreeBSD filesystem (using the already developed layering) might be an feasible approach that requires less Linux API chasing while retaining the "native" FS advantage at the expense of some extra memory (bhyve) ?

From what I understand, with qemu-usermode you're still risking incompatibilities, as QEmu needs to be able to handle the syscalls to translate (or 'transmarshal', if that's a word) each of the syscalls.

There has been some work done: https://reviews.freebsd.org/D21570. It appears stalled, though.

There used to be a thing like this called jetpack [1], but unfortunately the project has died.

[1] - https://github.com/3ofcoins/jetpack

Maybe I'm wrong, but isn't this a key component required to have native support (without having to use a VM?)

Frankly, though docker failed to deliver most of its promises, but at least it became a standardized deployable unit.

macOS seems to do alright

Docker on macOS uses a VM for the kernel. Its much slower than on a Linux host

MacOS runs a VM with Linux installation on it. That's why it takes so long to start, it is slow and uses so much CPU power and memory to run.

Not to mention the unusable volume performance

Reply on news.ycombinator.com