ZFSBootMenu – A boot loader to manage ZFS boot environments for Linux

This capability was something of which the lack on Linux has long puzzled me. Solaris actually implemented a very early incarnation of this ability (called "live upgrades" at the time from its original use case) back in the early '00s- in Solaris 8, and on top of UFS no less, if I recall correctly. It evolved over the next decade first adding ZFS into the mix, then finally morphing from the early "live upgrade" stuff into the full "boot environment" concept around 2010 with Solaris 11. FreeBSD implemented it around 2012, in the early days of their ZFS work. More than a decade ago. That puts Linux at least ten years behind the curve here, and arguably closer to twenty.

I'm a fan of using the right tool for the right job, and jumping freely between Solaris (or OpenIndiana nowadays), Linux, and FreeBSD for any given deployment is par for the course. Until now, all other things being equal, FreeBSD or Solaris would often win out if minimizing downtime* was a much higher priority than ease of replacing admins. Assuming that BE support in Linux matures quickly, that calculus has now swung strongly in Linux's favor.

*Re: minimizing downtime, if somebody is puzzled as to what I mean, think of the last time that you had a Linux installation fail to come back up to full operation after a borked round of package upgrades. It's not often, but it does happen occasionally. Now imagine that the time you spent getting back up and working, whatever it might have been, was reliably less than sixty seconds. Now imagine it's 2am, you're not even fully awake following a panicked phone call from the operations night shift, and your job hangs in the balance. Makes quite a difference.

My NAS has long been on OpenIndiana. Boot environments mean zero risk OS upgrades. At one point I could have gone back to a 4 year old OS version and booted it with no data loss.

You can create one at any time, so it brings an even better take on VM snapshots to the physical machine world. Hacking on something and want a fallback? "Beadm create beforehacking" and you're safe.

TBH, this is really cool. I liked that I'm able to choose snapshots for booting - a very good recovery option. The interface is well polished, and comes with fzf for quick searching. It's a true dream for distro hoppers, since ZFS works like thin-provisioned partitions (though distro options are limited due to ZFS). Pretty cool in and out.

But it turned out to be a super-overkill for me. Firstly, I stopped dual-booting like a decade ago. I run everything else in VMs. Secondly, the host system these days hardly breaks. Lots of things work out-of-box unlike old days, and service settings can be isolated in containers. Thirdly, my host environment can be recreated within an hour including the download time and few trims, as long as `/home` is backed up. So I don't worry much about the root partition.

I wonder how this is working for others.

The btrfs subvolumes are structured like this:

<root subvolume>/$(hostname)/${environment}/@volume (e.g. @root, @home)

snapshots look like this:

<root subvolume>/$(hostname)/${environment}/volume_$(date -u +%Y-%m-%d_%H-%M-00)

I have a few scripts "make-snapshots", "backup-snapshots", "update-shell", and "update-commit". make-snapshots creates readonly snapshots of my system, backup-snapshots does incremental backups of those to my NAS, update-shell creates a writable snapshot of @root as @root-update and drops you into a chroot environment. You can then run all the portage commands you want without fear of borking your current environment. Upon exit it checks whatever the /usr/src/linux symlink points to, copies the associated vmlinuz and initramfs images to the EFI partition, and creates/updates a boot entry in rEFInd. You can then boot either into your previous version or the update version. Once you're satisfied that your new environment works, you run "update-commit" which deletes the @root subvolume and replaces it with your current @root-update subvolume.

A change I've been considering is to drop the concept of having a @root subvolume at all. Current implementation requires two reboots: one to get from @root to @root-update, where (if it's good) you delete @root and make a writable snapshot of @root-update as @root. The second reboot is to get onto (the new) @root. An alternative might be to include the date/version in the name of the writable snapshots as well. "committing" the update would just mean setting the current booted subvolume as the "head". Future snapshots/updates will be made from that subvolume. No need for a reboot because you're currently on it with everything mounted correctly. Any writable subvolumes older than "head" would be cleaned up upon booting "head".

Could even go a step further and add something to my initramfs where if you try to boot a version where the writable subvolume has been deleted, it would make a temporary writable subvolume for it from the snapshot.

When we talk about sorry state of REAL tech evolution this and many others features should be counted...