FreeBSD Jails the hard way
clinta.github.io
clinta.github.io
My home server did originally run with jails which I'd created by hand and that worked really well for a while. But I started running into issues keeping the jails up-to-date. So I recreated those jails with ez-jails and the whole platform has been faultless ever since.
In fact it's been such an effective set up that I've been considering flattening my Proxmox box and running FreeBSD + Jails on that as well
There are other problems, but those are the most prominent.
As for why it's not gained more momentum, that would likely just be because Linux popularity dwarfs FreeBSD. Much the same reason as why so much effort was made to port ZFS to Linux despite FreeBSD already having excellent support. But for what it's worth, jails have been widely used within the FreeBSD community for quite some years.
It's a great shame FreeBSD doesn't get more love because it really is a fantastic platform. But then it's just one of a long list of very credible operating systems that often get overlooked.
I don't buy this, why would someone who complains about the consolidation and standardisation of core tools/services in Linux want to move to FreeBSD which does exactly the same as it is a full operating system and only supports what they ship, anything else you are on your own, just like what we get with standardising around systemd across distros.
The choice/flexibility which is potentially lost if the entire Linux ecosystem embraces systemd was never available on FreeBSD in the first place, what standardising around systemd is doing is moving Linux distros closer to one of the often touted advantages of choosing one of the BSD distros, which is a standard set of core tools/services which can be targeted (however in Linux's case it is across distros as opposed to the BSD's where it is per-OS).
#use syslog-ng instead of the built-in syslog
syslog_ng_enable="YES"
syslogd_enable="NO"
You can set Storage=none and ForwardToSyslog=yes in /etc/systemd/journald.conf [0].
[0] https://wiki.archlinux.org/index.php/Syslog-ng#syslog-ng_and...
(This sort of inability to make decisions about what tools you use for your system is one of the complaints about systemd)
To add one more anecdote: give voidlinux a try at some point. It uses runit as its init system and boots at least twice as fast as a systemd-based arch linux system. As with startups, the idea is less important than good execution, and systemd only succeeded on the former (so far).
With all that being said, I do like that I know how to do common administration on both Arch Linux and RHEL and Debian without learning three toolsets.
Honestly, why is this important? I want a system that is fast once booted, not a system that boots quickly.
That's not my experience, in the discussions I've seen and taken part in, most (as in vast majority) of the detractors are opposed to systemd as a consolidation of a large array of core tools and services all under the same 'umbrella', which is exactly what the BSD's do at an even higher level as they are developed as full operating systems.
I'm not sure what 'happiness to introduce bugs and unbootable systems' refer to, I've been using systemd ever since Arch made the switch (~4 years ago). I've yet to have stability issues and I haven't read of any widespread problems in the Arch forums either.
There are (and likely always will be) distros which default to and promote alternatives (Gentoo comes directly to mind), but in the grand scope we're seeing a huge convergence around systemd across distros, which tells me this type of standardisation is something the Linux ecosystem really pined for overall.
That combined with the systemd developers saying 'Systemd is a Linux-only project, we'll only be accepting patches for Linux and we'll only be supporting Linux. If this breaks on non-Linux systems, you're on your own.' Granted the community has adapted very well, as Gnome Shell running on OpenBSD demonstrates, but the non-inclusive attitude must have left a sour taste in a lot of people's mouths and you can understand why they feel bitter.
I'm just not sure that I believe the systemd team is said team. Their behavior on mailing lists and bugzillas has been extremely disconcerting at times, such as when they screwed over kernel developers who used the debug kernel option and basically said 'Tough shit that we broke the existing standard workflow, we don't care' ( https://bugs.freedesktop.org/show_bug.cgi?id=76935 )
Whether or not you agree that reasons like that are enough to distrust the team, if I were to keep running Linux in production, and make my own personal choice about what I am using, I can no longer realistically do this. Extremely necessary things such as udev being merged into systemd (and being a large pain in the ass to compile and use without systemd) mean this is no longer something as easy as replacing one or two components.
I'm all for convergence. Hell, I don't even particularly like sysv init - if systemd was a product as refined as even SMF, I'd care a lot less. And, I like the ease of use of journalctl - I spend a lot less time mangling text with grep and awk and sed when going through logs when I'm not on a system where things are being sent to splunk or kibana.
Whatever happens in the Linux community, I'm not particularly affected. SmartOS and OmniOS already have things very similar to systemd because of their Solaris heritage, and they work well. The BSDs seem to be moving in that direction, and I expect they'll create a solid product. I'll use those and be happy to do so. When I have to deal with other environments where Linux is used, well, I've learned plenty about systemd and feel perfectly comfortable working with it. I won't particularly like it, but ultimately, it's not something I care about to do more than discuss it on the internet.
- still unfixed shutdown errors from systemd - new shutdown errors from systemd - failed to boot or shutdown a system due to a systemd bug hence fixed - harder to fix if shutdown or boot fails compared to old init systems
My impression is that there isn't enough QA for the amount of churn they have in each release. Most bugs are very obvious and not someone using an obscure configuration for his sparcstation.
Standardization for a single common administration interface is a good thing but the resulting product is of sub-par quality thus far.
Could you go in to some detail here, what are the tools that have been turned in to monolithic binaries ?
The systemd project consists of some 70 different binaries, the vast majority of them being optional, and each of them handling specific tasks very much in the UNIX tradition.
They are being developed in the same tree, exactly like the BSD's, and just like the BSD equivalents they want to make use of features the kernel can provide even if that makes it less portable.
Even without that, systemd is still technically unsound in its design: http://blog.darknedgy.net/technology/2015/10/11/0/
I missed this - any link you'd recommend to catch up on what happened?
https://lists.freebsd.org/pipermail/freebsd-announce/2016-Ja... http://www.breitbart.com/tech/2016/01/01/freebsd-community-b... http://blog.randi.io/2015/12/31/the-developer-formerly-known... https://lists.freebsd.org/pipermail/freebsd-advocacy/2015-Ju... https://www.freebsd.org/internal/code-of-conduct.html
A fair statement would be "FreeBSD may eventually adopt a systemd-like approach."
And what do you mean by a systemd like approach?
"etc/rc.d is quite sophisticated for what it does, but the paint on /etc/rc is obvious"
"Too many things need to know explicitly about dependencies"
"Even the Linux die-hards have essentially grasped the necessity of systemd"
Talk is by Jordan Hubbard, FreeBSD co-founder (https://en.wikipedia.org/wiki/Jordan_Hubbard)
Discussion at https://news.ycombinator.com/item?id=8645188, video at https://www.youtube.com/watch?v=Mri66Uz6-8Y (49 minutes)
(The FreeBSD core team appears to strongly dislike all the currently proposed options.)
It furthermore provides a FreeBSD jails extension to the systemd unit file syntax. The JailID=i setting for a service unit is translated into "jexec i" in the native service bundle's run program.
Being able to docker pull/commit/push is what made docker so popular with developers. The "alternatives" all seem to lack this. I almost see it like I see dpkg vs apt-get. You can do what you need with a pile of deb files and dpkg -i, but why, when you can simply apt-get install? Same concept.
Obviously there's always room for improvement and people's experiences often differ, but I've honestly had no major issues with hardware support in the ~15 years I've been running FreeBSD.
The very first VPS provider[1] based their offering on jail and FreeBSD. They still offer them today (I sold the business in 2004).
We didn't call them "VPS" - that term came about later, in 2002 I think ... we called them "server instances".
We also didn't have any of these nifty deployment/monitoring/management tools - there was nothing but jail(8).
[1] JohnCompanies in 2001.
On the other hand, I associate Docker more with the machinery it builds on top of containers. It's got all the snapshotting and build/distribution infrastructure. I think popular wisdom is also only to run a single process in a Docker container, whereas most people I've seen (myself included) treat jails a lot like a virtual machine.
Jails are really popular amongst FreeBSD users and I use them a lot myself, but FreeBSD is nowhere near as popular as Linux, so that's one thing that held it back. Also, I think a big part of what has made Docker popular is all the deployment and distribution stuff on top of bare containers. The whole Docker philosophy of mostly immutable containers and microservices was what people wanted and FreeBSD jails aren't really intended for that.
Jails are engineering based.
It may make no difference to you, but it is the same difference between religion and science.
You don't hide problem behind nice easy tools easily deployable, you do it the hard way.
And as BSD says: jails are strong cages, but they are made of glasses, and when they break, it breaks badly.
Engineers don't believe in reassuring technology, they control them and do not fear the ugly details, they are aware of the domain of validity, accept imperfections and do not hide them under stickers, propaganda and success stories. They do risk control, they right manuals and point out weaknesses.
Jails user at least know this solution is a brass bullet and can fail you.
And sometimes good users worth more than good technologies.
A good soldier is valuable more than good weapons.
That's why early adoptant of linux are going to BSD. To get the control back and go back to basic engineering approaches that are stable.
I learnt in this process that you can learn from the mistakes that made linux fail the community : evangelization, consensus, philosophical approach, over-promising and being nice.
Jails are in the system.
FreeBSD has had jails for a long time and people using FreeBSD in production have been using them for a long time.
Linux just has a lot more attention.
With Linux being GPL, shipping a product using it means the driver code pretty much has to be released (unless you go to the lengths that Nvidia has done to keep a shim between the kernel and their GPU code).
I'm just now diving into docker. Docker is really about the images. Docker can run on lxc or in kernel namespaces. And there is ongoing work to port Docker to FreeBSD where Docker images will run in jails.
But apparently it took Docker image format and registry to make it popular enough among developers to make contanerization the old new big thing.
This is kind of an odd statement. All lxc is is userspace tools for managing namespaces and cgroups. If you're using lxc, you're still running on kernel namespaces
I tried to use zones many years ago..
For "isolated services" I mostly ran things like debian VMs with 128M of ram and 4G of disk space. In theory zones could do that too, but solaris was fundamentally incompatible with the idea of running a tiny 'container' with only the software that you needed installed.
As for your comments regarding Solaris, I was using Solaris Zones about a decade ago and, inversely to your experience, I found the whole process really easy. In fact it was on Solaris where I first learned about OS containers and which really convinced me to adopt them to many other projects over the years since.
To be honest, it is really weird reading your comment as my experiences couldn't differ more. If your memory is up to the job, I'd be curious to read more detail about the issues you had.
So the way I used to use debian was I was using xen. If I needed something like a tftp server, I would bring up a tiny VM with nothing but the base debian system installed. Then I would simply apt-get install tftp-server-whatever (via puppet) and call it a day.
Since all it was running was sshd+syslog+postfix(bound to lo) and the tftp server, patching was easy. I didn't have to deal with managing updates for 2000 packages that I was not using. I could also upgrade each vm separately.
Zones did not work that way at all. You could do a 'sparse' zone, which would inherit some directories from the host system, or whatever the opposite of that was that would just go and install a full 4G+ solaris system into the zone. When I complained about the fact 'single use' zones had to have 4G of software installed, I was told to just use clones, which did not use extra disk space.
The problem is those full clones then needed to be patched.
It was basically impossible to create a small single purpose zone that did not contain 4G of software and was easy to manage.
The sparse zones were not usable because then you could not install or update software in isolation. I see the same thing on the ez-jail page:
> Because the basejail's copy of the userland is shared by the other jails, updating the basejail automatically updates all of the other jails. Either source or binary updates can be used.
So, if an update to basejail breaks something, it breaks every jail linked to it? What if I want to test my software on FreeBSD+1?
Really, the problem wasn't the zone technology, the problem was the horrible user-hostile solaris userland. This is why Triton appeals to people.
As would the VM's you said you ran previously. Sadly patching is one of the biggest problems in IT. Which is where tools like Puppet come into their own (which I think you did say you run?).
> "The sparse zones were not usable because then you could not install or update software in isolation. I see the same thing on the ez-jail page:
> > "Because the basejail's copy of the userland is shared by the other jails, updating the basejail automatically updates all of the other jails. Either source or binary updates can be used."
I think that's a little deceptive. You can still install packages in the containers separate from the basejail. ez-jails would be completely useless if you couldn't. basejail is essentially just the FreeBSD core (ie it excludes /usr/local).
> "So, if an update to basejail breaks something, it breaks every jail linked to it? What if I want to test my software on FreeBSD+1?"
Then you'd have that as a container without basejails inheritance. Nobody says you have to create every jail using the same templates (though you did earlier complain about having to patch everything in isolation :P)
Personally, I tend to just take a ZFS snapshot and then rollback if I ever run into issues. It's a little lazy but works well enough for planned maintenance windows.
-----
Those arguments aside you do raise some interesting points; and I hadn't heard of Triton either. So I've definitely learned something new today. I can't really say I've had the same requirements to spin up and destroy new containers with the same frequency that you have, which would explain the differences between our experiences. For what it's worth, ez-jail does have a solution to those kinds of usages via "flavours" (loosely like VMWare VM templates). However I suspect this might still fall a little short of your specific requirements.
> (though you did earlier complain about having to patch everything in isolation :P)
The problem wasn't so much that the zones needed to be patched, the problem was that they were full installs of solaris. Solaris didn't really have a concept of a 'base install', at least not without a ton of debugging and manual dependency management [0]. This meant that the only easy way to install a working solaris system was to all the packages [1]. This meant that you then needed to patch all the packages.
I think some of that got better in solaris 11(?), but I never made it that far. I think I switched to nexenta and then to debian/kfreebsd.
[0] I tried a minimal install, I got the kernel to panic by trying to enable nfs4+krb5p without having some separately packaged crypto kernel modules installed.
[1] Unless you were a solaris admin. The ones I asked said it was possible, but that I should just install everything.
It's not as obvious or well documented how to set it up this way, but it was maybe a couple days work, including my first rc script ever, and the script to assemble the jail in our environment (you could probably have a tar file with the contents of the jail, and just add the conf/data files)
If you know how to build a minimal chroot, you know how to build a minimal jail, and then you just need to configure it, etc... If you don't know how to build a minimal chroot, check out option 3 'most minimal chroot possible' here: http://ram.kossboss.com/debian-ubuntu-build-minimal-chroot/
It's written against Debian Linux, but the same thing basically works for FreeBSD, except ldd on FreeBSD omits the loader (/libexec/ld-elf.so.1), but I sort of remembered that from somewhere, and I used truss to see what system calls were being used before it died.
I ended up writing my own very simple jail wrapper [1] (~200 lines of shell) which does most of this in a very simple way - I wouldn't advise this for general use but the code is simple enough to understand what is going on behind the scenes. I chose not to use jail.conf for my use case as I wanted to dynamically create/destroy lots of jails programatically and it was easier to just call jail(8) directly in the case.
[1] https://gist.github.com/paulchakravarti/9afea9e889bc992dbdd2...