Introduction to Immutable Linux Systems
dataswamp.org
dataswamp.org
The FCOS devs introduced a new feature called CoreOS Layering which lets you define your system in a Dockerfile and FCOS will rebase to that state and all you have to do is reboot to configure your server. It is super powerful.
Anyways, your next project needs a VM, give it a shot. I made a Python based CLI tool to help you develop locally on a Linux workstation to create a Butane file to fit your needs. Below is the GitHub for Bupy and a good example of running an app (Paperless NGX) on FCOS with the CoreOS Layering features.
https://github.com/quickvm/bupy
https://github.com/quickvm/fcos-layer-paperless-ngx
https://coreos.github.io/rpm-ostree/container/
https://github.com/coreos/enhancements/blob/main/os/coreos-l...
Do you have any thoughts you’d like to share on flatcar as the other project with CoreOS lineage?
As for me my main difficulties have been figuring out what to do with these projects in a bare metal environment. Building VM images is cool, but much of the time I want to do things like install to an existing drive or even onto a ZFS pool underneath.
As for building VM images, I don't actually do that in my setup. I just use the base FCOS image, boot it with a barebones Butane to configure disks and then use the CoreOS Layering features to setup my workload.
If you want to use ZFS on your setup, check out https://github.com/coreos/layering-examples/blob/main/build-... which has an example of building the ZFS on Linux module so you can setup your ZFS pools.
How hard is it to install CoreOS on a Raspberry Pis? Some installation guides on the Internet look quite complex...?
But I haven't actually tried CoreOS on a pi yet, could be interesting.
I think that VanillaOS and SUSE are working on similar things, but we're not an OS project, just a downstream from Fedora. Fedora's full support is underway but with what's already working perfectly our methods are already IME some of the most robust and easy ways of delivering Nvidia drivers for example.
How is it different from what you call “layering”?
Legit question, just trying to understand why you feel it’s an advantage.
Adding packages in an image has the benefit of pseudo-reproducability (have the same image on multiple computers) and the added robustness of your base system being built elsewhere daily. Your computer just pulls the diffs. For example, there have been issues with rpmfusion on Fedora that ublue users completely avoided. Codecs & other essential rpmfusion packages are included in the images, and the rpmfusion repository is removed after they are installed. This way, if something package-related breaks it breaks at the image build stage, and an ordinary user wont even notice it before it is fixed.
The most noticeable benefit IMO, though, is being able to ship the same changes on top of a base image every day for multiple machines. This is not only packages, but for example udev rules, and other QoL things like our `justfile`s, configuration for https://just.systems/ that has some useful scripts for adding the kargs necesarry for Nvidia drivers to work and `just update` for updating the system, flatpaks & distroboxes.
I think the bills are paid by Jorge (the kind of "founder" of the project), at least I think so, though some of the other top members with jobs in the Linux/Cloud world might be helping. Donation paths and such have been considered, but the bills aren't too huge so nothing has been rushed in.
For the registry, GHCR serves us entirely for free. No egress costs, no ingress costs, nothing. No plans to change providers, and I don't think they have plans to raise pricing either. We could probably find an alternative host pretty easily, though, through the cloud contacts and knowledge some of the devs here have.
The VMs booted from images. The image and the mutable differencing disks were entirely in RAM (although user profiles were on spinning rust). A desktop host for 25 users would boot to accepting remote logins in about 4 seconds.
Least painful Windows system to patch.
Weird phrasing. Haven't seen that before.
It is OK to not know stuff. It is not OK to argue that because you didn't know something lots of other people do, that the thing is is obscure or nonstandard or weird.
Just because you've been exposed to that term does'nt mean it's as widespread as you seem to think it is.
You remind me of people I would meet in one state, who would use certain slang or have certain customs particular to that state and insisted it was a USA wide custom, despite having never left their state.
I very quickly get the impression you are one of those "need to have the last word" types, so I won't be responding to you further.
Cheers.
I would go so far as to say this is the standard informal/slang way to distinguish rotating magnetic media from optical/solid-state media, and off the cuff I'd say it has been since the advent of consumer optical media about 30 years ago.
A definition from a decade and a half ago:
https://etherealmind.com/network-dictionary-spinning-rust/
From 8Y ago:
http://www.dba-oracle.com/t_spinning_rust.htm
It's in Wiktionary:
https://en.wiktionary.org/wiki/spinning_rust
And Urban Dictionary:
https://www.urbandictionary.com/define.php?term=Spinning%20R...
I'm not, but I appreciate your confidence in asserting so.
Finding evidence of that term being used has no bearing on how widespread that term was. Just because you've encountered it doesn't mean it is as widespread as you seem to assume.
You remind me of people I would meet in one state, who would use certain slang or have certain customs particular to that state and insisted it was a USA wide custom, despite having never left their state.
I very quickly get the impression you are one of those "need to have the last word" types, so I won't be responding to you further.
Cheers.
In my (limited) experience with something like Silverblue, the base system can be configured but when you start adding applications (like say, Firefox), it is lacking when it comes to configuring that because you're using Flatpak and I don't know of a way to tell it to both install all Flatpaks I want along with all of the configuration.
I guess there's some way of installing Flatpaks en masse and then dotfiles can take care of the rest?
https://universal-blue.org/tinker/mindset/#resist-the-urge-t...
Have you tried Impermanence?
This helps keep the (I agree, awful) configuration portion of nix minimal.
Until these immutable systems support the stacking of custom overlay filesystems as a first class feature^1, people will continue to run mutable systems.
1: to account for use cases the developer can't or won't support.
In practice this is one of my favorite things about nix. I find that I contribute much more to open source in general because it’s so easy to say “replace this dependency with my version when you build this package”.
https://rebeccaskinner.net/posts/2021-06-06-nixifying-a-haky...
This sort of thing is very common with GUI software on Linux. Outside of this bubble, most software comes with batteries included. Eg Solidworks never makes me download optional dependencies, but FreeCAD made me do it literally every 15 minutes as I move to the next step in a CAD/CAM/sim/render workflow.
Also see https://www.joelonsoftware.com/2001/03/23/strategy-letter-iv...
> it’s never the same 20%. Everybody uses a different set of features. In the last 10 years I have probably heard of dozens of companies who, determined not to learn from each other, tried to release “lite” word processors that only implement 20% of the features. This story is as old as the PC. Most of the time, what happens is that they give their program to a journalist to review, and the journalist reviews it by writing their review using the new word processor, and then the journalist tries to find the “word count” feature which they need because most journalists have precise word count requirements, and it’s not there, because it’s in the “80% that nobody uses,” and the journalist ends up writing a story that attempts to claim simultaneously that lite programs are good, bloat is bad, and I can’t use this damn thing ’cause it won’t count my words. If I had a dollar for every time this has happened I would be very happy.
Certainly, Google Sheets didn’t do that. Nor has airtable (valuation considerations aside, it has seen some adoption).
In a new category, or a new market, there is no benchmark. So that’s the east case. But sometimes new products do displace old products, and sometimes just by distribution or pricing at the cost of features.
So how do you reconcile that, Joel?!
Also worth mentioning that Sheets has competition in this space now that in-browser 365 Excel is free. Tho neither is feature-complete vs desktop Excel
"Sorry boss, been sorting out my init.el"
FreeCAD might not come with enough battery to power a full workflow, but I have also seen entire engineering firm not able to work without a particular AutoCAD plugin.
At that point why not just use a mutable system? This reminds me of how a decade ago everybody rushed to move to nosql and then immediately reinvented schemae in their projects.
Compare Haskell: both Haskell and C support both mutable variables and constant ones. But the ecosystems and idioms are very different.
How it is done can be seen with OBS: flathub has several OBS plugins available (com.obsproject.Studio.Plugin.*).
A problem that's less well-solved is situations like Cantor, where a single application is a front-end for multiple executables. To get it to actually work you'd have to package Scilab, Sage, Maxima, Octave, R, and Julia in the flatpak itself, and these are non-trivial programs to package. There are workarounds with flatpak-spawn, but at that point why not just install the application natively? (I think the "right" answer is to set up a dbus service for like "system-octave" or whatever and have a separately flatpak'ed interpreter register it, but as aesthetically pleasing as this solution is it doesn't seem to have induced me or anybody else to actually do it...)
Installing any number of packages, then removing them in any order, at any future point(s) in time, is equivalent to never having installed them at all.
This leaves some distros out, but I feel like it’s the important part of the concept.
Have a look at 'uniquely represented datastructures' and 'History Independent Data Structures'.
You'd need to take special care to make sure that your block device (eg SSD) allocates blocks independent of history.
Beyond that, consider other changes, like say you decide to change DNS implementations and then later decide to change your default DNS server. If you uninstall the provider to change back to previous one, do you also want to change back to the old server you were using or do you want to retain that change? Personally, I'd want to only change the provider but keep the new server.
Then consider difficulties with directories shared across multiple machines. Let's say you have /home/${USER} set up as an NFS or Samba mount so you can keep the same files across multiple workstations. If a program respects XDG config dirs and stores stuff in there and you uninstall on one workstation, should it remove the files from all of them? Do you want all your devices to be identical or do you only want the home directory to be identical? There is no possible way for a package manager on a single system to know this.
I currently use Ubuntu for my server, and I see there's ostree in my repos, I haven't gotten around to trying it yet but I'd like to just start versioning my system as is with it. If that's too difficult/not possible to do, then I intend to switch my server over to Silverblue at some point because I really like the idea behind ostree.
The other reason is snapshots, because no system is perfect. An operating system should in fact be designed around the fact that something will go wrong. Windows has had this for years, and some linux users have done it with btrfs.
But Fedora Silverblue does it seamlessly with ostree and grub, so if something does go wrong in a newly applied update, you simply choose the last working one in the grub menu and go on with your day until you have time to resolve it.
This for me to use Fedora as my daily driver is crucial.
For example, I was using a script written by an ubuntu user that was looking for a library with the name/location ubuntu puts it in. Fedora uses a different name for the library. My instinct here is to create a symbolic link with the Ubuntu name that points to the Fedora rpm managed library. Instead, to get it to work I forked the script, got it to build locally, made it look for either library name, ran local test cases, submitted the code as a PR upstream, etc etc etc. It took something that would normally cost me 1 line of shell to fix to something that took 90 minutes.
Even before I switched to Silverblue I was known to create Ubuntu containers to run tools from there.
That's what you should have done, run it in a container. If you're not comfortable with containers then the workflow in Silverblue will feel very strange.
I sympathize, but also AIUI this is exactly the sort of monkey patching that these systems are trying hard to avoid; yes, it fixes your immediate problem, but it leaves an undocumented, unmanaged change in how your system finds libraries. In my personal experience, this is the kind of change that leads to machines growing weird behavior that ends when I give up and so a clean reinstall.
1. With Fedora Silverblue 39 the most straightforward way is to add a Dockerfile layer just which makes your changes directly to the base image.
2. You can also create your own RPM and make the changes there.
3. Best is to actually run your script in an Ubuntu container (distrobox, toolbx, or podman).
Does that put the onus on you to rebuild container image in order to receive system updates?
How do you keep track of your custom modifications to /usr or /bin without using the package manager? Do you record them somewhere for reference?
Is this one of those things designed for career sysadmins/system builders only?
https://opendev.org/starlingx/apt-ostree is one such effort to bring ostree to debian.
You might find their forum (intended for end users, not much about development) or some of their repos useful:
I'd be awesome if there was a community effort around bringing the layering functionality and bootc enablement to all of Debian and then we could have our cake and eat it too!
I read a quora answer that estimated the Windows OS development budget at around 18 billion dollars, based on salaries. Imagine if Red Hat invested 2bn into Fedora to make it the Firefox of the desktop OS world. Just a 10% share is very significant against Microsoft.
They've come so far with so little, on the back of thousands of open source packages. That money could be used to keep those projects alive, and to sponsor them while they're being developed. Red Hat employees are already involved in a lot of them.
Redhat could invest 200bn into Fedora or any other project, and still people won't switch to it because even if it's "better" by some arbitrary metric, it's not what people are used to.
It's definitely possible, but it looks like Linux is more likely to gain popularity via WSL than via Fedora or similar.
And most of those people will use Windows at home, too, if for no other reason than to not have to learn how to use two different desktop systems.
It's so light, you can spin up VMs, one for a mail-server, one for a database, one for a firewall/router, each in a couple of seconds.
Tinycore is itself immutable, so you add a vdisk with a "package" and some config, mark it read-only, and job done. A single Virsh script handles the startup and shutdown of "services" - each being a Tinycore instance. Fun, and robust so far, but not sure if I'd put it into anyone's production just yet.
It’s solid and simple, unlike these other immutable Linux distros
I haven't yet needed to boot back into windows! If it stays like that for the next 6 months I'll cut over to Linux permanently and wipe the windows partition.
In the case of Nix, it sounds like it's more focused on reproduce-ability? It sounds like I should be able to take the Nix configuration file, plop it on another computer, and get the same system (except, perhaps, for /home).
Some of the others sound more like existing tools that provide snapshot/rollback capability, just with different implementations.
* system upgrades aren't done on the live system
* packages changes are applied on the next boot
* you can roll back a change
That's a atomic transaction, like a database. Although having to shut down the system to do a commit is a bit much.
Microsoft put atomic transactions into their file system years ago, but file system transactions were never used much. You'd like to have an install system where all changes commit all at once, and if anything goes wrong during install, nothing commits and you roll back to the previous state. In theory a transactional file system could do that. In practice, there's probably too much other non file systems state involved.
> 4.3. Facts §
> - NixOS / Guix are doing it right in my opinionBasically, nixpkgs is still the best way to run something on NixOS
Flatpak is an ugly hack that mixes together the completely unrelated tasks of packaging and sandboxing, while not being particularly good at any.
I would love if there was a seamless way to launch a distrobox os with a separate user home that could not touch my host system.
Likely a Real Hard Problem, but even some isolation would probably be an improvement of running everything under the same user account.
How can anyone adopt an unpolished hobby project that only seems to be tested on the dev's Arch box is beyond me.
If you want hard separation you'd need a VM. It'd be awesome if you could just --firecracker on distrobox create and get that. :D
Any good blogposts on developer workflows on Silverblue?
I can understand wanting an immutable system for a server, as it will likely cut down on maintenance, but for personal use... that just sends shivers down my spine... As someone having to support other (especially not very savvy) programmers when it comes to tool usage and their environment I hate to imagine having to deal with someone who'd want that kind of setup.
But even with "escape hatches" programming in container is very painful and uncomfortable. I've only ever seen this done by people who chose to work on a very restrictive system (perhaps for its appeal to their aesthetic feelings rather than any practical concerns). Or, maybe, their employer has bad IT, which both strictly enforces the rules and creates rules that acutely inconvenience the employees. In either case, it's a big hit to productivity. But, in some cases, there weren't much in terms of productivity to begin with (the programmer was bad with or without good programming environment), so the losses are imperceptible.
I've been prototyping some developer workflows with friends here: https://universal-blue.org/images/bluefin/developer-experien...
So far the major patterns are vscode with distrobox, vscode with devcontainers, vscode with devpod, jetbrains toolbox thing (which just runs everything out of the home directory, the OS doesn't care).
And then devbox/nix and homebrew in ~ is also an option if you're into that.
We didn't "just discover" some secret only known to the embedded community, lots of people have been working toward this exact goal for a long time, because we already knew for a long time that it has a lot of advantages.
no, you can just mount overlayfs with ram-backing for example
Basically, it's an image based OS that configures everything from a single config file on boot. https://docs.vyos.io/en/latest/introducing/about.html
> We could say that a Linux LIVE-CD is immutable, because every time you boot it, you get the exact same programs running, and you can't change anything as the disk media is read only. But while the LIVE-CD is running, you can make changes to it, you can create files and directories, install packages, it's not stuck in an immutable state.
This meant the mess systemd created each boot, could be dropped into the ram-drive with zero impact on the OS image. Effectively turning any Debian based system into a read-only OS backing image, but retaining the ability to boot into a normal writable system with a single boot flag.
This trick is a lot less finicky these days. =)
MicroOS Desktop turned into Kalpa (KDE) and Aeon (GNOME), but the latter has all the momentum.
Honestly tying the OS to the desktop environment is the only reason I haven't installed microos. Making a derivative with my preferred DE is on my "maybe someday" list, but that's a long list...
1. create a ram-backed filesystem
2. copy `/`s contents to that new filesystem
3. Optional: Unmount `/`
4. Mount the ram-backed filesystem on `/`
You can easily do that from the initramfs during booting. I'm applying this patch to the roofs created by debootstrapping Debian Buster. You can use the system just fine and make changes as you please. But when you shut it down, it's all lost. Everything I want to keep (like the permanent storage this system makes available over sshfs, NFS) is on seperate disks anyway. Sure, you need enough RAM to hold the entire rootfs (1.2G in case of Debian Buster) and it increases boot time a bit. For server applications, I don't care at all.
--- a/usr/share/initramfs-tools/scripts/local 2021-11-05 12:50:23.541088057 +0100
+++ b/usr/share/initramfs-tools/scripts/local 2021-11-05 13:02:14.483203576 +0100
@@ -180,9 +180,20 @@
# Mount root
# shellcheck disable=SC2086
- if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then
- panic "Failed to mount ${ROOT} as root file system."
- fi
+ #if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then
+ # panic "Failed to mount ${ROOT} as root file system."
+ #fi
+
+ mkdir --parents /tmp/diskroot
+ mount -t ${FSTYPE} ${roflag} ${ROOTFLAGS} ${ROOT} /tmp/diskroot
+
+ mount -t tmpfs -o size=6G none ${rootmnt?}
+ chmod 755 ${rootmnt}
+
+ cp --force --archive --verbose /tmp/diskroot/* ${rootmnt}
+
+ umount /tmp/diskroot
+ rm -r --force /tmp/diskroot
}
local_mount_fs()In my observation (and in datasets that I have access to), computers systems tend to follow the "infant-mortality" curve. This means that if they run for a little bit, they're likely to run for a long time (and in addition, if you have many of them, they tend to die around the same time). My conjecture is that many computer systems have initialization routines which are not as thoroughly tested as the normal operating state of the system. Due to this, we tend to run into more issues in "immutable" systems than you otherwise would in "mutable" systems.
It's not like you don't have all the same issues with deploying your long-lived mutable systems -- you're just feeling the pain less frequently, and postponing all the work to debug them until you're in e.g. a disaster recovery scenario.
Currently this is supported on Triton DataCenter only, but our internal roadmap has us building a standalone version similar to how folks use SmartOS standalone.
Silverblue/Sircea gets all the attention these days, but Endless is the oldest OSTree-based user distro by a long shot, and it’s still actively developed by the Endless Foundation.
It’s also the one most suitable for non-technical users. Definitely worth considering for that use case, particularly for very young users since it now includes plenty of tutorial content intended for that audience.
That's basically what I want, encapsulation and buttons :)
If you mean how would you do it yourself, you can use the tooling used by real distros. I'm not sure what tools they all provide, but Archiso (https://wiki.archlinux.org/title/Archiso) is probably the simplest to understand and modify because it's purely shell scripts.
In the past I’ve tried running a small USB drive in rented bare metal w/ diskless alpine, but the machine seemed to reboot randomly IIRC.
- upgrades/changes are atomic
- it's very clear whose responsibility it is to track what state, i.e., what parts of the filesystem are snapshotted and when
- it's easy to revert to any snapshot at boot time
- you're unlikely to have 'gaps' where you wish there was a snapshot but there isn't one
- snapshotting applies to configuration changes as well as package (un)installation
As it happens, snapshotting with automatic reversion is an approach some take with NixOS to better ensure that its configuration management is comprehensive. You can force all of your persistence requirements to be explicit by reverting to an old snapshot on boot.
Blog post which (afaik) first presented this idea to the community: https://grahamc.com/blog/erase-your-darlings/
Common implementation for NixOS: https://github.com/nix-community/impermanence
Presentation on that implementation from NixCon 2023 (just a couple weeks ago!): https://www.youtube.com/watch?v=QtBouFMyrWg
Fully statically linked.
Depending on the implementation, a system may offer more features. But this list is what a Linux distribution should have to be labelled "immutable" at the moment."
Immutable. I do not think it means what you think it means.
> 4.1.1 you can roll back changes if something went wrong.
> 4.1.2 transactional-updates allows you to keep the system running correctly during packages changes.
Last time I needed to roll back was OpenSSL in Ubuntu 18.04 and that was on one system.
I don think I've ever had a problem that 4.1.2 solves.
I don't want to have yet another Linux OS to solve a problem that happens once a decade.
If I wanted an immutable OS, I'd use a container OS on which I'd run apps as containers. Oh, wait, I already have that.
Just like Wayland, they're pushing it for other reasons (breaking existing stuff, locking you in to the GNOME desktop, etc), and using spurious reasons to promote it.
I purchased a 7900XTX on release day, with the Linux drivers being in a fairly rough state and new fixes being added daily. So, for the next 3-4 months, I was running Fedora Rawhide with the Koji repo added - about as bleeding edge as it gets, short of building locally from source. Rolling back definely came in handy once or twice.
Once things stabilized, I rebased back to non-Rawhide Fedora 37 and stayed there. Then, a few weeks ago, I found out that AMD had been working on ROCM for the 7xxx series, so I've rebased to Rawhide again to play around with AI tools on the 6.6 kernel.
I've also occasionally encountered non-critical bugs, suspected it might have been fixed upstream, and temporarily rebased to Rawhide just to check out if that was the case before reporting it. Pretty nice.