Advantages to running FreeBSD as a server operating system | Hacker News Reader

Advantages to running FreeBSD as a server operating system

klarasystems.com

66 pointssigna1151 comments

My favorite part of running a FreeBSD cluster was that you almost never had to reboot it. The kernel rarely changed -- most kernel settings could be changed on the fly with a config file, and most everything else ran in userspace. We had boxes with uptimes over 700 days.

Nowadays that matters a lot less, since you most likely have redundant hardware. But back then (in the 90s), hardware was expensive and more often than not you had one big beefy box running your entire website, so reboots mattered.

I miss running FreeBSD. But for a long time you couldn't run it in EC2, and I never ran it as a home server, because at home I'm more interested in experimental things that usually show up on Linux first.

But it was great what I was running enterprise on-prem hardware.

May be it's time to reboot Yahoo 2.0 along with its server infrastructure which were all running on FreeBSD.

For that matter, you could almost never reboot linux.

Linus supports kernel live patching, so you can get the latest bugfixes for the current kernel version without a reboot.

Anything else, including systemd, is upgradeable and restartable like a normal userspace component.

I really wanted to like FreeBSD. ZFS is cool, jails are cool, pkg has such a wide variety of packages that it's comparable to Arch.

In a world before Docker and btrfs, this would be enough to make installing FreeBSD worthwhile. I tried it anyway, for a home server, and it was mostly painful.

Jails are tricky and require third-party (?) tools and scripts to manage them effectively. And they don't give you much that Docker doesn't already. And, although it's not as bad as Arch, FreeBSD does update quickly and if you aren't frequently updating your server you can be left behind; it's not noticeable until you try to update and a bunch of packages break, or your jails are no longer running the same major version as their host and that causes subtle bugs.

I recently replaced my FreeBSD home server with a basic Debian machine running a docker-compose cluster, and it's so much simpler. I wanted to like FreeBSD. At its core, it's probably a better-designed system. But it requires too much additional knowledge and maintenance for something that Linux and Docker do just as well.

For me it's the opposite really. Every time I touch Linux, I'm overwhelmed by how messy it is. I spent hours upgrading an old Ubuntu box serving as a Ubiquity UniFi Controller and ended up in some kind of dependency hell. While trying to fix all that, disk space ran out. Then I had enough, whiped the whole thing and in 5 minutes installed the FreeBSD base system. A handful of binary packages and a couple of edits to config files later everything worked perfectly. It all feels so clean, with a minimal number of processes running.

My experience dealing with Linux over the years can be summed up by the exclamation "oh my fucking god can you people just PICK ONE."

There are certainly arguments for why the variety of replacements for simple, old, well trod and tested system tools are improvement and yet my 20 year BSD installations get on mostly fine without them and I'd be hard pressed to describe what they're missing in practical terms other than very specific stacks like Docker.

If you live in devops I can see that being a big deal but for the service functions themselves there is something to be said for leaving minor things at "good enough."

Running out of disk space isn’t the operating system’s fault

> Then I had enough, whiped the whole thing and in 5 minutes installed the FreeBSD base system. A handful of binary packages and a couple of edits to config files later everything worked perfectly.

There seems to be some confirmation bias here. Is writing an iso, installing an OS, installing a “handful” of packages, and configuring everything not somewhat equivalent to the “messy” alternative of upgrading Ubuntu? I bet they were very close to the same amount of work, but one might have seemed like more, or more daunting, due to your experience and preference for bsd. Just throwing it out there

As for Ubuntu upgrades, apt-get update, apt-get dist-upgrade, and do-release-upgrade have always worked well for me, though I hardly ever do a major version upgrade and opt to do a fresh install after the first minor point release. If you were jumping major versions or a few major versions, of course packages are going to give you a hard time. I imagine a fresh install might have gone as seemingly smooth as the bsd route you took. Anyway, hacking is hacking, keep at it!

You know, 10 GB should be plenty for what that machine is supposed to be doing. On Ubuntu, initially it was, but after several upgrades (despite cleaning package caches), not any longer. FreeBSD takes less than a fifth, and I know that isn't going to change much. Am I desperate to save a few gigs? No, it merely illustrates how much of everything there was that I didn't want there to be. Now the output of top fits in my small console window. It's about feeling in control.

Surely experience must be factored in, but I consider it a fact that Linux is less coherent and more prone to changes. Evolving quicker can be a benefit, yet for my server applications it simply never is.

> FreeBSD does update quickly and if you aren't frequently updating your server you can be left behind; it's not noticeable until you try to update and a bunch of packages break, or your jails are no longer running the same major version as their host and that causes subtle bugs.

In modern times, point releases tend to be about once a year, which doesn't seem too frequent to me? But if you use pkg, and the new point release added a syscall, you will have problems installing new packages (or upgrading) once the package builders update; I don't quite remember the timeline, but it's usually a few months after the point release. (Although, I think there was a time where pkg would default to a faster updating repository, and you might see problems sooner)

My upgrade process is a bit laborious (freebsd-update -r ... upgrade; freebsd-update install; shutdown -r now; freebsd-upgrade install; pkg update; pkg upgrade; reboot). Might be another pkg command I forgot, but it will likely prompt you.

If you're not interested in upgrading everything together at once, ports is probably a better choice. Sometimes after an OS upgrade, you'll need to install a compat port to continue to run old software; it's possible to do that with pkg, but IMHO, it's difficult to get the compat package installed without upgrading other packages, and then you may as well upgrade everything.

Personally, I don't find docker-compose simple. Probably from lack of experience, but I find it hides so much that makes it much more difficult to debug. Most of the time, I'd rather run everything in a single host, but I'm encouraged to use docker-compose by my workplace, so...

> shutdown -r now

I know this is in lots of textbook examples but I didn't think anyone ever actually typed this. I've always used 'halt' or 'reboot'

reboot and halt have a history of being less graceful on some systems, and I've been bitten. For FreeBSD, I think reboot skips shutdown scripts, doesn't notify all terminals, etc; just kill everything, sync the disk and reboot.

That's not ungraceful, but it's not exactly pleasant.

OTOH, my openwrt device doesn't have a shutdown binary, just a reboot binary.

Using 'halt' or 'reboot' won't run all the scripts needed to shut down cleanly, the correct command to use is 'shutdown -r now'.

IIRC, 'reboot' is aliased to 'shutdown -r now' in several modern distros

I use "shutdown -r now" instead of "reboot" because "halt" or "reboot" is not universally available, but shutdown mostly is.

> [...] But if you use pkg, and the new point release added a syscall, you will have problems installing new packages (or upgrading) once the package builders update; I don't quite remember the timeline, but it's usually a few months after the point release. [...]

The previous release in the branch gets dropped after ~3 months, probably aligned to the end of the month that its EOL lands in; then pkg builders switch to the newer release in the branch.

I find your assessment odd, since Docker itself is basically a third-party tool to manage cgroups, namespaces, and filesystems. I suppose it's another in a long line of what are essentially marketing failures on the part of FreeBSD...

FreeBSD jails don't have a one-command way to install a preconfigured jail for a specific service. And they don't have a way (that I know of) to write a config file describing a bunch of jails and services, then run that file and install all of those services on a new machine.

Docker is less elegant than jails in a lot of ways, but not having to figure out how to install each service on your own is a huge advantage.

I tried using Ansible to set all of my services up, and it really didn't work. Probably because I was using a different machine as the Ansible controller, and there is no Ansible plugin to connect to jails on a remote machine. I tried writing my own, and it just barely worked, but with a lot of bugs I could never fully track down (because Ansible's internals aren't documented at all). Eventually it just wasn't worth it to keep trying to use FreeBSD.

It has both, but like Docker, they don't ship in base. ezjail was the tool of choice for many years, and cbsd (which also manages the virtualization capabilities) has emerged as a mature replacement for the past decade or so.

I'm not disagreeing with your assessment -- nobody knows better than you which software you want to use! -- so much as listing options for anyone reading.

But I find it interesting how far ahead of its time FreeBSD was on this, and how badly it appears to have fumbled this ball. The official documentation still recommends ezjail, which is not good. Just another example, I guess, of "writing the code isn't enough."

What’s wrong with ezjail? I still use it, works great

> FreeBSD jails don't have a one-command way to install a preconfigured jail for a specific service

FreeBSD does have that tool, its BastilleBSD: https://bastillebsd.org/

For example, this is the Bastillefile for running consul: https://gitlab.com/bastillebsd-templates/consul/-/blob/maste...

> And they don't have a way (that I know of) to write a config file describing a bunch of jails and services, then run that file and install all of those services on a new machine

Sounds like you want to use pot with nomad: https://klarasystems.com/articles/cluster-provisioning-with-...

To add on, CBSD may also provide of these functions. So there are options.

[1] https://github.com/cbsd/cbsd

Honest question: what do you mean with “docker-compose cluster” ?

Would also like to know what this is.

Docker compose allows you to specify multiple docker containers all running on a single host. It makes setting up multiple services on a single piece of hardware relatively simple and repeatable.

> In a world before Docker and btrfs, this would be enough to make installing FreeBSD worthwhile

FreeBSD is really missing out in not implementing something to run containers, and it shouldn’t even be impossible.

The podman people basically reimplemented the docker daemon command line (and apis), and I remember a talk by Bryan Cantrill about how joyent reimplemented the docker API to spawn Illumos/SmartOS zones backed by zfs volumes (i guess either datasets or zvols). And there was some person reimplementing the docker apis using jails as a backend… but I guess that project never went anywhere and didn’t get the attention (and funding/developer time?) of the freebsd foundation.

Jails are tricky and require third-party (?) tools and scripts to manage them effectively.

Docker is a third party tool. People use third-party tools and scripts to manage them effectively.

Having done jails for years and now having to get comfortable with Docker for $DAYJOB, there's not a huge difference in complexity, just different complexity, as long as you actually learn the tools.

FreeBSD is great, if your hardware is supported. And unfortunately, that's a giant IF.

I wanted to try jails on a 2011 HP DL380 G7, which is enterprise level hardware. I had to jump through several hoops to even get off the ground. Apparently FreeBSD didn't come with power management out of the box. So I had to learn about powerd, Intel Speedstep, and CPU C-states (but hey, it comes with a mail server pre-installed). And even then I was having issues. So I gave up and installed Alpine Linux, which worked perfectly with no extra config. I learned LXD instead.

Since then, I've replaced that behemoth of a server with a couple low power, tiny desktops running Debian (bookworm has LXD in the default repos now). I didn't actually look, but I'm fairly certain FreeBSD wasn't going to play nice with Intel's QuickSync and GPU passthrough to jails like Debian/LXD does.

Anyways, that's always been my biggest hurdle with FreeBSD. I do like the cohesiveness of all their tooling though.

If you’re running lxc containers on debian you might as well switch to proxmox (it’s debian underneath), cluster those machines together and have a nice web ui, http apis and a cli.

If only it had an official terraform provider, it’d be perfect.

Thanks for your suggestion. It's a good one that I might consider in the future. I ran normal VMs in Proxmox for years, so I'm quite familiar with it. But I feel like having two separate systems gives me a more flexible playground for my homelab.

I want articles to get to the point quickly. I don’t want them to start with “What is FreeBSD?” I know people do it for Google SEO, but it’s messed up, please stop it.

There You go:

- https://vermaden.wordpress.com/2020/09/07/quare-freebsd/

So FreeBSD supposedly has full Linux kernel emulation or something to that effect. To the point where you can run Debian in a jail without virtualization? I am wondering if it is possible to rig up Docker to work in that?

Because I really adore jails but I agree that the tooling around them makes a mess of config and such. (I have been using ezjail which I'm told is now deprecated)

With so much of my world buying into the Docker cargo cult it would be nice if I could play along and I'm wondering if Linux-in-a-jail is a possible answer.

You may want to check out iocage for managing jails. It has not had a release in a few years but it is maintained.

https://www.freshports.org/sysutils/iocage/

Where is it maintained? The port is, but the upstream?

I have been looking to migrate to something else but my fear with iocage is that I'm going to be in the same situation a few years from now.

What's the current hotness? Bastille?

> I am wondering if it is possible to rig up Docker to work in that?

Unlikely, 99.9% the answer is no (at least not currently).

FreeBSD does not have “full kernel emulation”. What it has is a table mapping linux syscalls to freebsd equivalent, essentially an adaptor. This, along with a typical linux runtime (glibc and other libraries) means that you can execute linux binaries.

But this goes as far as the translation table goes.

To run docker containers you’d need an equivalent syscall to create (pid, network, uts etc) namespaces and (cpu,memory) cgroups.

There is no equivalent of those syscalls in FreeBSD at the time of writing, as far as I know.

I evaluated using FreeBSD for a project. The biggest challenge for me was I couldn't figure a way to fit modern DevOps into FreeBsd. OCI compliant containers are not supported. The best way I could come up with for a minimum blue-green deployment strategy was writing a ton of scripts myself. I can not imagine running a fleet of FreeBSD servers where a team of developers deploy 10+ times a week to a swarm of containers running micro services.

What solutions do other practitioners use with FreeBSD? Or do you abandon modern DevOps practices entirely and hold a meeting to determine partition sizes, provision servers to individual services running on specific named computers and plan for a software upgrade for weeks in advance of a deployment?

If you are working with FreeBSD in the cloud, you can use Terraform to provision on AWS and GCP. If you have to manage bare metal FreeBSD systems at scale (hundreds of systems), most devops folks use Ansible. There are playbooks out there for this. It's not too painful to work with.

If you are looking for Kubernetes on FreeBSD, don't. For on-prem Kubernetes you could deploy on Linux virtual machines on FreeBSD servers, managed using Ansible. But it's never going to be as fast as Linux. I wouldn't use it in production.

The best answer is to leverage FreeBSD for what it's good at : bare metal services like databases, build environments, file servers, and networking devices. And co-manage your deployment with Linux (Kubernetes) using Ansible/Terraform.

I used FreeBSD till early 2000's, when Linux had become a better choice.

I've installed Chromium on FreeBSD 12 and it was impossible to enable to antialised fonts, no matter what I had tried. Generally, it is worse as a desktop system than more even obscure OpenBSD.

> Generally, it is worse as a desktop system than more even obscure OpenBSD.

This is just untrue.

> Generally, it is worse as a desktop system than more even obscure OpenBSD.

Only because you couldn't get AA with chromium? FreeBSD is my daily driver, stable as a brick. Everything just works.

Never going back.

I run NetBSD as a desktop and AA also works with Chromium. This looks strange.

> Generally, it is worse as a desktop system than more even obscure OpenBSD.

there is a very polished, BSD-based desktop available, albeit at a fairly high price compared to FOSS alternatives

MacOS?

this mostly seems like cheerleading for ZFS? how about ASLR support, the amazing pf tables firewall, dtrace awesomeness, cool things to do with jails, or bhyve virtualization?

granted ZFS is a big flex on the BSD official site as well since its effectively the only way the OS can do things like raid10 with dedup. instead of porting XFS or BTRFS the BSD devs chose to keep developing a fork of Oracle/Sun's commercial product.

> instead of porting XFS or BTRFS the BSD devs chose to keep developing a fork of Oracle/Sun's commercial product.

XFS doesn't even try to solve the same problems, and BTRFS is GPLv2 and only just starting to lose its reputation for not eating data, and that only if you avoid some RAID mode. So yes, ZFS is far away better

AFAIK XFS has been part of FreeBSD for a long time. I remember around 2008 at least.

https://man.freebsd.org/cgi/man.cgi?query=xfs&sektion=5&apro...

Zfs is the same cosebase that runs on linux (freebsd people literally dropped their implementation and imported openzfs into their source tree). It’s not merged in linux purely for license incompatibility (cddl vs gpl, iirc?)

For the other stuff (aslr, pf, dtrace etc) linux has either equivalent or better alternatives (eg: perf/ebpf, iptables/nftables, kvm) etc.

Regarding btrfs… most linux distro ship it and some (eg: fedora) use that as default fs.

Zfs is more intuitive and more ergonomic in my opinion, but i might switch to btrfs anyway if anything so I don’t have to worry about updating the kernel.

> instead of porting XFS or BTRFS the BSD devs chose to keep developing a fork of Oracle/Sun's commercial product.

Again, not true and actually the exact opposite: since openzfs is cddl-licensed, oracle cannot use that in solaris.

Both BTRFS and XFS write support is supported by 'sysutils/fusefs-lkl' on FreeBSD. No need to port anything.

ZFS is superior to BTRFS - first of all its safe and secure - I do not know how many more years wil ltake BTRFS to finally properly fix their RAID5/RAID6 implementations to not loose data .... and that also gives warning for general usage in any other way.

In which way iptables/nftables is better when IPFW or PF?

The only feature that KVM has against BHYVE is live migration ...

The FreeBSD Handbook

tl;dr:

1. ZFS 2. pkg 3. unclear

... profit?

Reply on news.ycombinator.com