Are Immutable Linux Distros right for you?
linuxblog.io
linuxblog.io
There’s also been a ton of very advanced development in the space. You can now take bootable containers and use them to reimage machines and perform upgrades. Extend your operating system using a Dockerfile as you would your app images:
Yes, but a majority of the remaining 0.0001% are people on Hacker News, so it's going to get discussed here!
What? No they're not. I literally have never once in my career seen a server deployed with an immutable distro.
plus, given your wealth of experience, I'm more interested in what linux you run on your personal laptop
Most of my time is in Tmux anyways. Over the last 15 years the client side has been one of MacOS, ArchLinux, Ubuntu, and now Windows/WSL2.
The real activity has shifted up to the orchestration and service discovery layers - nomad, k8s, consul, and whatever fleetmanagement/cluster management layers maintains the hosts (a lot of homebuilt + terraform + chef/argo-workflow in our world). It's been years since I was really that concerned about the linux host side of things - why care about "immutable" when the entire machine/image is ephemeral for < 1 week (or in some cases, <1 day) anyways?
If you really think about it, what's the difference between spinning up a VM with a preconfigured image and spinning up a VM with an _immutable_ preconfigured image?
Divergences from the base image are inherently limited because of that.
Of course there's many more Linux servers out there than there are programmers, but the OS the programmer uses to develop on is just as important as the OS they deploy to.
Nowdays Linux is very stable, you can have things that don't work properly but you won't need a full re-install.
Every time I’ve tried to run a standard Linux distro like Ubuntu for more than a couple of years I inevitably end up breaking something in a way that I can’t recover.
Are you taking snapshots to roll back to?
If you install the newest versions of whatever from random repos or compile stuff yourself you are very likely to mess things up. But nowadays there is very little reason to do that. And you can pick a distro that releases at a pace you are comfortable with, so you have choices.
Usually broken distro upgrades I see are because people run "curl randomdomain.ck/totallysafescript.sh | sudo bash -" to install things or use custom repos.
That `totallysafescript.sh` could at least be inside of the package manager scope. Most of the times someone already did it, and published it to AUR.
IMO the reason why there are so many people running random scripts in Ubuntu/Debian is due to how more difficult/inconvenient it is to get a dpkg .deb when compared to a PKGBUILD file. Same for MacOS, in which you have to either rely on Homebrew wizardry or just running the script
The AUR is still not as good as proper package management and shouldn't be considered a stable or reliable method of software distribution at scale.
I've been running rolling release distros for a decade and never had any problems - you have to follow some software migrations when needed, but I managed to migrate to systemd on Arch without an issue while any dist upgrade on ubuntu was wrecking my system.
It's not a good distro. I don't know why people insist on using it. Notice that the GP said Debian instead. (Probably Stable, because testing and unstable will break within 10 years.)
My gut is that if I was to try and get my 3 monitor setup such that the seams are all pixel aligned, I would be in for a world of pain. I imagine that would be the same for other OSes, as well?
I liken Nix to source control in the time of CVS. We need two more implementation iterations before its going to be useful to the general public.
Such an exaggregation. Many thousands of people use it by choice, despite all alternatives. Hardly a disaster, by any definition.
But among those who do, there are plenty of people who have learned Nix well enough it's no longer a weird arcane thingy that spews out incomprehensible errors for them. Although, I guess, among those no one will deny Nix can be better (but there are no multi-billion-dollar corporations spending tons of their resources on it).
It's like vim. First time you run it you probably can't even exit it - so, of course you think it's a disaster ;)
Is that an aggregation of exaggerations? :p
Actually, that portmanteau kinda-works in this context.
When it works, it’s great. I like that I can install (and uninstall) much of the software I use declaratively, so I always have a “clean” base system that doesn’t accumulate numerous little packages at strange versions over time in the way that most workstations where software is installed more manually tend to do.
This is a trade-off, though. Much is made of the size of the NixOS package repository compared to other distros, but anecdotally I have run into more problems getting a recent version of a popular package installed on my NixOS workstation than I had in probably a decade of running Debian/Ubuntu flavoured distros.
If the version of the package you want isn’t available in the NixOS repo, it can be onerous to install it, because by its nature NixOS doesn’t follow some popular Linux conventions like FHS. Typically, you write and maintain your own Nix package, which often ends up similar to fetching a known version of a package from a trusted source and then following the low-level build-from-source process, but all wrapped up in Nix incantations that may or may not be very well documented, and sometimes with a fair bit of detective work to figure out all the versions and hashes of not just the package you want but also all its dependencies, which may in turn need packaging similarly themselves if you’re unlucky.
It’s also possible to run into this when you’re not installing whole software applications, including those that are available from the NixOS package repository, but rather things like plug-ins for an application or libraries for a programming language. You might end up needing a custom package for the main application so that its plug-in architecture or build system can find the required dependencies in the expected places when you try to install the extra things. Again, this is all complexity and hassle that just doesn’t happen on mainstream Linux distros. If I install Python and then `pip install somepackage` then 99.9% of the time that just works everywhere else but frequently it won’t work out of the box on NixOS.
It’s one of those things that is actually perfectly reasonable given the trade-offs that are explicitly being made, yet still makes NixOS time-consuming and frustrating in a way that other systems simply aren’t when you do run into the limitations.
This comment is already way too long, so I’ll just mention as a footnote that NixOS also tries to reconcile two worlds, and not all Linux software is particularly nicely arranged to be managed declaratively. So in practice, you still end up with some things being done more traditionally/imperatively anyway, and then you have a hybrid system that compromises some of the main benefits of the declarative/immutable pattern. There are tools like Flakes and Home Manager that help to overcome some of this as well, and as others have said, they are promising steps in good directions, but we’re not yet realising the full potential of this declarative style and it’s hard to see how we get from here to there quickly.
I think this is my favourite comment about Nix ever. I'm not going to stop using Nix until a genuinely better alternative arrives, but that day can't come soon enough
Fixing the issue ends up being rather difficult though
A custom GPT have been surprisingly helpful after feeding it manuals for nix, nixpkgs and NixOS and other Linux books.
I can make a nix-shell for each project but then every nix upgrade was forcing me to go through a lengthy reinstall + wrecking compatibility sometimes.
Not to mention the amount of derivations I had to write myself just to use latest packages.
Using things like virtualenv instead of nix-shell can fix the general instability, but packaging is too big of a problem.
I went back to Arch.
Containers, and snapshots+clones are your friend. For a while I was doing ZFS snapshots and clones of Gentoo userlands.
However, if you knew how bad things really are with glibc and how not-well designed Linux is to resist badly behaving software, and how easily some big players can inject badly behaving software into the channels you are fetching from, you would probably seriously consider Qubes.
illumos is a kernel you can rely on to run somewhat arbitrary software.
NixOS makes that too, its whole "system output path closure" is as immutable as every single store unit within it. But NixOS "reuses" units which are unaffected by NixOS config changes when applying new config, making its "system rebuild" super fast and light on resources when something like a single config file is changed in NixOS config. And possible to be done "in place", unlike with "conventional immutable distro"
Saying it’s just about integrity is like saying docker images are just about integrity… they absolutely are not. They give you atomic units of deployment, the ability to run the same thing in prod as you do in dev. Many other benifits.
In my understanding, integrity is exactly about software being in certain known correct state, including absence of anything which is not part of that state. Of course integrity of parts of software system like individual packages contents does not make it really reliable when the whole system does not have it. NixOS has it and also allows to "run the same thing"
Isn't that exactly the point of NixOS?
It also means you can completely leave out the package manager from the target machines, as it’s only used to bootstrap creation of the single deployable unit. Implementing that bootstrapping step is where nix and friends are helpful in this setup.
Nix offers the same guarantees. The point was that you don't need the whole system als a single unit (e.g. an image). A system can also be a tree of immutable output paths in a store (where a single output path often, but not necessarily, corresponds to a package).
In that model a system is basically an output path in the store (in reality it's a bit more complex) and has other output paths as transitive dependencies.
Upgrades/downgrades are atomic, because they just consists of selecting a different output path that represents a system. Upgrading creates a new output path that represents a system (either by downloading from a binary cache or building that output path) and booting into that output path. Rolling back consists of booting into the previous output path that represents a system.
There’s all sorts of ways a feature like that could provide value. Adding atomicity to system package managers would be a large, obvious win.
Transactional filesystems still sound cool, but Unix filesystems already have enough features to implement atomic package installation on top of them.
Yes, but it’s pretty awkward. You need to do the final commit via a rename or symlink. It would be far more convenient if you didn’t need to do that.
I think the issue is the posix filesystem API. Nobody writes better filesystems because no software would use the new features, and no software supports fancier filesystem features because the posix API doesn't expose them.
It'll probably take someone like Apple or Google to fix this. Similar to 16kB pages.
People tried: https://en.wikipedia.org/wiki/Transactional_NTFS and https://learn.microsoft.com/en-gb/windows/win32/fileio/depre...
"[Transactional NTFS (TxF)] was introduced with Windows Vista as a means to introduce atomic file transactions to Windows. It allows for Windows developers to have transactional atomicity for file operations in transactions with a single file, in transactions involving multiple files, and in transactions spanning multiple sources – such as the Registry (through TxR), and databases (such as SQL). While TxF is a powerful set of APIs, there has been extremely limited developer interest in this API platform since Windows Vista primarily due to its complexity and various nuances which developers need to consider as part of application development. As a result, Microsoft is considering deprecating TxF APIs in a future version of Windows"
It was tried by one specific team, in one politics-guided environment. It was bounded to a ton of unrelated features because of politics, it was bounded to a specific API because of politics, it was bounded to a timeline and a team because of politics, and it was about a completely different OS.
We have no idea even how it went, because we can't trust the people reporting about it are talking about the correct thing.
"ZFS is one"
"It's not really because it was written by a team, and because it was made for Solaris which is a different operating system, and there was lots of politics involved with its licensing, and because it was bound to the POSIX file API, and because it does complex volume management as well, and because it happened within a timeline before Sun went out of business, and we even have no idea how ZFS went because every documentation and article which says it's about ZFS might be talking about something else".
uh huh.
- Learning. Figuring out how to migrate a setup even to the most mainstream-like immutable distro (fedora silverblue) can take a while, and to niche distros like talos even longer. However, a k8s-friendly setup with low customization requirements would help to speed up the migration (but it requires more powerful machines).
- Long term support. Regular distros like Debian and AlmaLinux offer free 5 and 10 year support cycles which means maintenance can be done every 1 or 2 years. On the other hand, immutable distros would require much more frequent maintenance, once every 6 months. A weekend every 6 months is a sizeable part of my time budget for hobbies.
One aspect in which immutables distros have improved a lot is in resource usage. They used to require significantly more disk space and have slightly higher minimum requirements than regular distros, but that doesn't seem to be the case anymore.
What's maintenance in the context if immutable distros? Running "ujust upgrade"? That's done automatically in the background for my Aurora installation.
Also, they're working on CentOS based LTS versions of Bluefin: https://universal-blue.discourse.group/t/call-for-testing-bl...
There are also smaller maintenance tasks that are tipically ad-hoc solutions to unsolved problems or responses to monitoring alerts. One of this ad-hoc routines was checking that logs do not grow too large, which used to be a problem in my first systemd centos, although not anymore.
PD: thanks for the bluefin read, it made me discover devpod/devcontainer as an interesting alternative to compose files
Intuitively, this seems opposite, because you could obviously 'mutate' (or mutilate) your Debian system until the updates break. Isolating user changes should make updates easier, not harder. Also MacOS uses a 'sealed' system volume and updates are like butter there.
Smooth as in "no data loss", sure. Smooth as in "supports the software I buy and use for long periods of time" is most certainly not true, even despite half the software for Mac being statically linked. Windows and Linux arguably do better at keeping system functionality across updates even with their fundamental disadvantages.
Maybe it would help in a datacenter
The high availability of ChromeOS is a good example of these advantages in a business of educational context.
If you update using an immutable distro, you rebase back on to your previous deployment or adjust a pin and you’re done. Immutable distros save you tons of time handling system upgrades, and the best part is you can experimentally change to a beta or even alpha version of your distro without any fear at all.
But that basically doesn't happen between release upgrades, not unless you're doing something with third party repos at least.
> If you update using an immutable distro, you rebase back on to your previous deployment or adjust a pin and you’re done
I genuinely don't know, but can you do security updates without rebasing? Just keeping some working version pinned sounds like bad idea to me, and doesn't even save you time because you'll need it resolve that problem eventually anyways.
Nvidia would like a word
The discussion in the article talks about using containers and flatpak and snap and all those things bundle dependencies and really swell the disk usage requirements. Is there a good solution other than owning a massive SSD?
It isn't as big a problem for servers which don't change as often and where you need instant rollbacks, but I'm using immutable (or atomic distros like nixos) on my laptop and having trouble.
It makes me think I'm not using these systems correctly.
Docker uses layered images, one on top of another for each step of the build process, to deduplicate in Docker, you try to reuse layers, but it's not perfect and having duplicate files is very very common.
Flatpak uses OStree, which has a content addressable file store, meaning files are stored and linked based on their checksum, so duplicate files only exist once and are linked into their respective locations.
There is work to make Docker use a system similar to Flatpak (see composefs).
Still a bit of an annoyance, but one I don't notice once it's set up.
In terms of installation size it's not a problem either, as one can verify using `flatpak list --columns=size,name`:
1.3 MB Flatseal
2.9 MB Extension Manager
47.7 MB Celluloid
604.6 MB Freedesktop Platform
680.0 MB Freedesktop Platform
533.8 MB Mesa
533.8 MB Mesa (Extra)
469.8 MB Mesa
469.8 MB Mesa (Extra)
46.9 MB Intel VAAPI driver
50.9 MB Intel VAAPI driver
20.3 MB FFmpeg extension with extra codecs
790.0 kB openh264
763.9 kB openh264
243.7 MB GNU Image Manipulation Program
7.7 MB HEIC
17.6 MB Characters
14.2 MB Connections
25.4 MB Image Viewer
946.7 kB HEIC
25.5 MB Sushi
39.8 MB Papers
941.3 MB GNOME Application Platform version 46
1.0 GB GNOME Application Platform version 47
794.1 kB Fonts
137.7 MB gThumb Image Viewer
1.1 MB adw-gtk3 Gtk Theme
269.6 MB Firefox
482.3 MB Signal Desktop
In short, storage isn't a problem at all for any computer produced in the last 20 years.
I switched to EndeavourOS. Between flatpak and brew and mise, I have relatively well sandboxed applications. This gives me most of the benefits of the immutable OSes, although nowhere near as rigorous, obviously. For a technologist, though, it's fine.
They just hit their first alpha release, but it has been under development for years already. They focus on rust-based tooling, so even their coreutils are the rust versions instead of GNU. I read the alpha announcement yesterday, and might give it a spin later next year.
So far I've been very happy with Kinoite. I upgrade the base system once a week, but everything is installed in my Arch based container, so updates are fast and do not require a reboot.
On my workstation I use the Aurora Linux, a spin of Kinoite with extra tools such as tailscale added to the base image. On that machine I haven't needed to use rpm-ostree at all.
I gave Aurora a quick spin before going back to Endeavour. Didn’t work well for me.
If one container breaks, just dump the list of installed packages and start over. The base system is rock solid.
You use something like distrobox to use a rolling release with regular package updates on the atomic core.
But NixOS is immutable in a very different way to all the mention distos, which are focused on containers, isolation, and layers; Maybe the author doesn't consider it to be in the same category?
Personally, I've decided that NixOS is not for me. The concept is great, but the actual experience seems to be held back by Nix (the language and the tool) being hard to understand and debug.
The problem with nix repl, is that it only seems to help if you already understand both nix and how the derivations are actually implemented in nix. It's pretty useless as a learning tool.
It's the same as learning Haskell. Outside of syntax and some basics, you don't need to have deep knowledge to use ghci. And Nix and Haskell are both just substitution-based evaluation (lambda calculus) which imo is 80% of understanding them.
For some reason, I have a really hard time groking Haskell, and Nix seems to fit in the same boat. I don't know why.
It's not the functional programming. I love doing functional style programming in Rust, Python and recent versions of c++. And I didn't have any problem with Prolog and Lisp for those few university courses. I have a suspicion that my brain just finds the concept of lazy evaluation by default to be deeply offensive.
What I do know is that personally: I could never grasp Nix; The repl didn't help; The repl was the extent of the debug tooling; I never found good documentation to help me learn; and I was getting anxious at the thought of doing anything on that server.
But with impermanence it can be effectively immutable.
It uses SquashFS images and layer them on each other. You can choose to save your modifications in a new image, or discard them. E.g. you can run a Puppy Linux from a CD-R (one time writeable), by appending all your changes.
I think that's a great model to be immutable, but AFAIK Puppy Linux doesn't have the convenient tools to manage these snapshots, switch between them, roll back and such, and they don't seem to go in that direction. (I used Puppy Linux as my default system for a while, but I lost touch with them and I don't know how are they doing now.)
I want full control over my system. Immutability means leaving part of that to the OS developer. Definitely don't want that. Even though it's ostensibly better for security (though it's only really making one step in the kill chain harder, which is establishing persistence).
Second, immutable distros are primarily a distribution and update mechanism that vastly improves the current model of running X number of package updates and scriplets on every machine and hoping it works. There’s nothing that stops you from remounting a filesystem as rw at least on any of the distributions that I know of. There’s also plently of stateful, rw holes for data and configuration on “immutable” distros.
But it's more the configuration that I want to be able to adjust, or to recompile things. As a typical example, on alpine I always need to recompile sudo as their standard version doesn't allow PAM which I need. On an immutable system such tools would usually be in the immutable part.
I had problems with macOS when they switched to immutable (and if you turn off the protection it turns off a whole load of other things too). If I as much as changed the /etc/ssh/sshd_config it would revert with updates.
And really the traditional package system works totally fine for me.
Immutable does not mean you cannot change it according to your wishes. It just means that each change must be explicitly declared in order to be included in the next system image.
In some ways, having a declarative, immutable distribution makes the process even easier, as is the case with NixOS. If you want to patch your sudo, it would be as easy as doing
security.sudo.package = { pkgs.sudo.overrideAttrs (old: { patches = [ (fetchPatch {url = "<url>"; sha256 = "<patch sha256sum>"})];})};
And then you're done.
100% truth be told, having a declarative, immutable distro has allowed me to experiment and configure my system way more than I would have otherwise. I mean, I can do anything because I have the safety net of rolling back if I mess up.
Furthermore, being declarative means I know exactly how I got to my end solution, instead of having to memorize a bunch of steps from different attempts that may or may not have been successful.
See Apple's implementation where the OS files are protected by signatures and the system won't boot if they're changed. Immutable does mean you can't change it, though I'm some cases you can enable and disable some parts. Nix is declarative and perhaps not immutable. It's advised not to mess with config files but you still can if you wish, it's just a bad idea because it'll be overwritten.
I'm not really against declarative management though I'd consider it something more appropriate for servers where I don't want to change stuff on the fly. On my workstation I don't want to do a complete change time every time I want to modify something. I also don't want to learn the complex syntax so I've never really dived into nix.
I like FreeBSD's compromise of having most configuration in one file but still a traditional system.
Yep. An attacker can just surreptitiously add a line to your .bashrc instead of modifying the base OS.
But they also use this to enforce DRM, for example if you turn off system integrity protection you can't run iOS apps anymore. This is exactly the kind of thing that bothers me about it.
If you gain root by editing .bashrc and replacing sudo or placing a file in .local/share/applications to replace an application that the user trusts (like the Settings app) to give their password to, then you can just inject your payload into the initramfs and get persistence.
I don't believe any desktop distro is signing and verifying the initramfs.
And yeah the problem is that the initramfs is built on the machine itself. So it would have to have the signing keys which defeats the purpose.
Apple does sign the entire boot process but they have the benefit of a strictly defined hardware set to support.
Windows can also sign the entire boot process, but they unlike Apple can't make the system folders read-only due to backwards compatability (Windows 10S experimented with this idea but was scrapped).
There's nothing stopping Linux distros from having a secure boot process, but the initramfs either has to be scrapped or pre-built by the distro.
Just brainstorming here.
What if the initramfs was rebuilt every time the OS was upgraded. During an OS upgrade, the user is asked to cold boot, the machine does a special boot, requests the user's disk decryption password, and uses it to build and sign the new initramfs, based on files signed by distro maintainers.
Then for every ordinary boot, immediately after disk decryption, we keep the disk decryption password in memory for just a little bit longer, and use it to check the signature on the initramfs before continuing with the boot.
The "signature" could be the secure hash of [the disk decryption password concatenated with the initramfs binary], or something (ask a crypto expert -- perhaps KDF+HMAC is better?)
I'm guessing the disk decryption password is much harder to steal than the user's root password?
(I might be totally out to lunch here, I know nothing about Linux boot. The above comment is written in the spirit of "learning about things by asking dumb questions"!)
On most distributions, they already are if the kernel changes in anyway. This is what ends up filling up your /boot partition.
Then for every ordinary boot, immediately after disk decryption, we keep the disk decryption password in memory for just a little bit longer, and use it to check the signature on the initramfs before continuing with the boot.
You can just reuse the existing architecture of secure boot/tpm to store public keys (although GRUB is a disaster and will likely break your boot chain of trust). You normally can't add additional keys unless the system is in setup mode. But, the process to add a self signed key is not user friendly and you'd still need to store the private key on disk to sign additional images, which malware can access.
There is a process to generate something known as a UKI, where it can be a signed kernel/initramfs verified by the firmware secure boot, but I believe only Red Hat has keys to sign anything and most distos get given a signed shim that will only boot the turd called GRUB.
My only real gripe is that Firefox still ships as an rpm in the base image. I understand that they want to include a working web browser at all costs, and I don't think they can distribute the Flatpak version with the base image, but it's annoying that I have to mess with the image (removing Firefox) to then re-install the (more up to date) Flatpak.
A cuda dev environment is a 'toolbox create' away
And it's Linux so regardless of the distribution you'll be dealing with some amount of weird shit on regular basis. Has been true since I cycled home with a stack of slackware floppies almost thirty years ago. There's always configuration files to fiddle with, weird shit to install, etc.
But an immutable base OS makes a lot of sense and it's not mutually exclusive with that being updated regularly. Containerization is the norm for a lot of server side stuff. Effectively, I've been using immutable server operating systems for almost a decade. It's fine. All the stuff I care about runs in a container. And that container can run on anything that can run containers. Which is literally almost anything these days. I generally don't care much about the base OS aside from just running my containers hassle free on a server.
Containerization would make sense for a lot of end user software as well. IMHO things like flatpak and snap would be fine if they weren't so anal/flaky about "security". Because they are protecting a mutable OS from the evil foreign software. Running a bit of software that needs a GPU isn't a security problem, it's the main FFing reason I'm using the computer at all. Or own a GPU. This needs to be easy, not hard. And it shouldn't need a lot of manual overrides.
If I run a browser or things like Dartable, I usually have no reason to run them in crippled/unaccelerated mode. Sorry that's not a thing. It's the main reason I bypass flatpak on Manjaro for both packages. And I bypass PAC as well because I trust Firefox to have a good release process. So, I use the tar ball and it self updates without unnecessary delay. Which considering a lot of its updates are about security is exactly what I want.
Same with development tools. I use vs code and intellij. Both can self update. I have no need for a third party package manager second guessing those updates or dragging their heels getting those updates to me.
For the stuff I care about and use every day I like the direct connection to the developers. Mostly repackaging adds very little value. If somebody finds a bug, they should be reporting it upstream; not providing some workaround. Most mature projects are pretty good about releasing, packaging and testing their software. The only reason linux package managers exist is the gazillion ways there are to package things up for different distributions.
It's great.
But I'll share my experience: I think an immutable / really is the way forward. Just the ability to roll back and boot using an older snapshot is great: I have had an update break the boot, but I have the option of running a single command to roll back while I investigate the issue. At the time the issue happened I was busy with life and I simply rolled back and used that version for three months before I had time to investigate.
Strictly speaking this does not require the current / to be mounted read only, but merely requires periodic bootable snapshots be taken and these are available to be used as a read-only /.
For some reason, immutable Linux distros seem to struggle with the idea that a single physical disk might contain both volumes owned by the distribution and persistent volumes owned by the user that are not managed by the distro. Last time I checked, Talos was basically unusable on a single-disk system if you want persistent volumes.
Sadly, most M.2 NVMe devices don’t seem to support namespaces, which would otherwise be a decent way to kludge around this problem.
Most old distros can do things like this with no particular difficulty. But the Kinoite installer (which is presumably the same as the Silverblue installer) is half-baked and buggy.
I wonder why embedded products like Nvidia Jetson do not come with an immutable Linux (and instead are based on Ubuntu which updates itself on every opportunity via apt and snap and whatnot).
Although many places start with Ubuntu or Debian in my experience it’s common to invest a lot of time and energy in getting out of that unmanaged setup once the company scales.
Hardware vendors in this space can’t be trusted, so you need to make sure the board is actually fit for purpose. Outside of the hobbyist space you have to be really careful. There are often business objectives that rely on the board working a certain way.
> UCI is the successor to the NVRAM-based configuration found in the White Russian series of OpenWrt.
Also the web UI counterpart, LuCI: https://openwrt.org/docs/guide-user/luci/luci.essentials
I've been running OpenWRT on my home router since ca 2017, and I found LuCI both quite intuitive, and immensely powerful. Simple things are simple, complex or difficult things are possible, with just clicking around.
Unfortunately if something can't be done with LuCI, you're pretty much on your own - the documentation for the internals is scarce and expects you to be an expert/developer.
Containerized apps are nice, containers for development is nice, but you can have those in "normal" distros with a lil work setting things up.
The real killer feature is you can have a bleeding edge system with zero fear of breakage.
I suppose TinyCore Linux in its default configuration also counts.
Maybe this will finally change.
Having said that, no, I don't see any usage of immutable Linux in my future.
My experience with Snap is that it bugs me about Firefox updates multiple times a day for two weeks. Okay, it does then update automatically and break the running program, but I can't claim to be unaware.
Apt is the thing that updates packages completely without my awareness, with Unattended Upgrades. Mostly it works, but I have to blacklist NVidia utilities, as they need to be in sync with the driver in use.
https://snapcraft.io/docs/managing-updates#p-32248-pause-or-...
127.0.0.1 api.snapcraft.io
127.0.0.1 archive.ubuntu.com
127.0.0.1 security.ubuntu.com
127.0.0.1 mirrors.kernel.org
127.0.0.1 deb.debian.org
127.0.0.1 ppa.launchpad.net
127.0.0.1 flathub.org
127.0.0.1 dl.flathub.org
Or configure them properly by disabling auto-updates, configure unattended-upgrades appropriately for your needs, and only update your apt packages from a known, internal mirror endpoint that doesn't change until you point it to a new timestamp.
That's how it works in the real world, in production. It's not 1994, we don't hack hosts files anymore.
Any particular reason?
Just wondering because it’s really just the system itself and my desktop environment that I find the benefits of immutability most pertinent, whereas it’s something of a bad fit for applications with the woes flatpak and friends bring for desktop integration and such.
Silverblue also supports package management via brew, which works pretty well for CLI utilities
> An immutable Linux distribution has its core system locked as read-only. This ensures the base operating system remains untouched during normal use, protecting it from accidental changes, unauthorized modifications and corruption.
So, in other words, I'm using an immutable system already! (Windows 11)
The system booted fine! It just didn't have a graphical desktop environment installed anymore. But it was up and running, not crashing or anything like that! It was no more 'unbootable' for lacking a GUI than the server hosting this website is. :)
But yeah rollbacks are a great way to handle situations like that, so it's a great feature for a package manager to have.
SteamOS is immutable and incredibly successful. macOS (not Linux of course) is also immutable and very successful.
As long as the OS’s have a concept of overlays, an immutable system rarely gives up much in the way of flexibility either.
Immutable Linux distros, esp. NixOS pull all kinds of shenanigans (ELF patching to begin with) to achieve what they want to achieve with a complete disregard how they change the behavior and structure of the system. What you get is something which resembles a Linux distro, but with monkey patches everywhere and a thousand paper cuts.
When a Linux distro can become transparently immutable, then we can talk about end user adoption en masse. Other than that, immutable distros are just glorified containers for the cloud and enthusiast applications, from my perspective.
I think there’s been well done immutable systems and it’s something that can be achieved with a mainstream Linux distro.
That's true, but these problems are worked on for a quite long time, and the core ethos of a Linux distribution is being able to be on both sides of the fence (i.e. as a user and as an administrator which can do anything).
For example, in macOS, you can't customize the core of the operating system since the eternity, and it's now sent you as an binary delta image for the OS, and you have no chance to built these layers or tweak them. This increases the ergonomics a ton, because you're not allowed to touch that part to begin with.
However, with Linux, you need to be able to change anything and everything on the immutable part, and this requires a new philosophy and set of tools. Adding esoteric targets like "I really need to be able to install two slightly different compilations of the same library at the exact same version" creates hard problems.
When these needs and the mentality of "This is old, let's bring it down with sledgehammers. They didn't know anything, they're old and wrinkly people" meets, we have reinventions of wheels and returning to the tried and working mechanisms (e.g.: Oh, dynamic linking is a neat idea. Maybe we should try that!)
Immutable systems are valuable, but we need less hype and more sane and down-to-earth development. I believe if someone can sit down and design something immutable with consideration to how a POSIX system works and what is reasonable and what's not, a good immutable system can be built. Yes it won't be able to do that one weird trick, but it'd work for 99% of the scenarios where an immutable system would make sense.
> However, with Linux, you need to be able to change anything and everything on the immutable part, and this requires a new philosophy and set of tools.
Taking macOS as a North Star as a successful mutable Os, most people don’t need to, and shouldn’t be, touching the immutable parts. I know the assumption is that you need to do that on Linux, but I don’t think a successful distro should require most casual users to go anywhere near that level of access.
If they do for some reason on Linux specifically, why would overlays not be sufficient? They achieve the same results, with minor overhead and significant reliability gains.
> I believe if someone can sit down and design something immutable with consideration to how a POSIX system works and what is reasonable and what's not, a good immutable system can be built.
But someone has done that. macOS is POSIX. SteamOS just works for most people.
First of all, macOS doesn't have package management at the core OS. You can't update anything brought in as the part of the OS from libraries to utilities like zsh, perl, even cp. .pkg files bring applications in to mutable parts of the operating system, and can be removed. However, OS updates are always brought in as images and applied as deltas.
In Linux, you need a way to modify this immutable state in an atomic way, package by package, file by file way. That'd work if you "unseal, update, seal" while using EXT4 FS. If you want revisions, you need the heavier BTRFS, instead, which is not actually designed for single disk systems to begin with.
You can also use overlays, but considering the lifetime of a normal Linux installation is closer to a decade (for Debian, it can be even eternal), overlays will consume tons of space in the long run. So you need to be able to flatten the disk at some point.
On the other hand, NixOS and Guix are obsessed with reproducibility (which is not wrong, but they're not the only and true ways to achieve that), and make things much more complicated. I have no experience with RPM-OSTree approach.
So, if you ask me we need another, simpler approach to immutable Linux systems, which doesn't monkey patch tons of things to make things work.
> But someone has done that. macOS is POSIX.
Yes, but it's shipped as a set in stone monolithic item, with binary delta updates which transform it to another set in stone item. You can't customize the core OS. Can you? Even SIP sits below you, and has the capability to stop you with an "Access denied." error even if you're root on that system. It's a vertically integrated silicon to OS level monolithic structure.
> SteamOS just works for most people.
SteamOS is as general purpose as a PlayStation OS or car entertainment system. It's just an "installable embedded OS". Their requirements are different. Building and maintaining a mutable distro is a nightmare fuel already, making it immutable yet user customizable is a challenge on another level.
It's not impossible, it's valuable, but we're not there yet. We didn't find the solutions, heck even can't agree on requirements and opinions yet.
I again disagree with your assertions that you need to replace the OS contents. You just need to be able to reflow them to other ones. That’s the macOS way, the flatpak way etc..
I think the issue is that you are claiming that you MUST be able to do these things on Linux, and I’d push back and say no you do not.
And your comparison of SteamOS to a console or in car entertainment is flat out incorrect. Have you never booted it into desktop mode? For 90% of users, what are you expecting they need to do in that mode that it’s not “general purpose” enough to do?
Yes, it’s not impossible. It’s been done multiple times , successfully as well.
Yes, but this doesn't make my point invalid and irrelevant. $PATH overriding is as old as this planet itself, and its primary purpose is not to make immutable systems more user-friendly. It's just a way to use your own binaries if you want to override itself.
Being able to use flatpak, etc. on a system doesn't and won't allow you to change the core OS. Being able to reflow things doesn't solve anything about system management, esp. on the long run.
> I think the issue is that you are claiming that you MUST be able to do these things on Linux, and I’d push back and say no you do not.
I'll politely won't move and say that it depends. If you're building appliances or embedded systems, that might be discussed if there's an absolute requirement to remove the update mechanism, but for a server and end-user system which can live more than a decade, this is a core requirement which can't even be discussed, full stop.
> And your comparison of SteamOS to a console or in car entertainment is flat out incorrect.
I don't think so.
> Have you never booted it into desktop mode? For 90% of users, what are you expecting they need to do in that mode that it’s not “general purpose” enough to do?
No, but even Valve doesn't say what you say. What Valve says is "Users should not consider SteamOS as a replacement for their desktop operating system. SteamOS is being designed and optimized for the living room experience." (emphasis mine) [0]
Its desktop mode is akin to Browser sessions present in consoles, or "Other OS" which used to be in Playstation. IOW, a nicety, not a feature.
> Yes, it’s not impossible. It’s been done multiple times , successfully as well.
Yes, to answer different set of requirements with set of different features. We still have a long way to go with "An immutable, daily driveable, desktop and server distribution which can be maintained, used and updated for a decade" (a-la Debian).
macOS is not immutable at all.
https://eclecticlight.co/2021/10/29/how-macos-is-more-reliab...
There are two main "daily driver" usability issues on SteamOS by default if you need to do technical work:
- Limited software availability via the flatpak repositories.
- Not being able to install certain programs as easily without needing containerisation of some kind (if that even solves the problem in some cases).
Distrobox solves a good amount of both issues on SteamOS, for coding work at least. Slap a virtual Ubuntu on and you're off to the races.
And steamOS is totally a general purpose OS, it’s just got a non-general purpose frontend it defaults to.
Uh... yeah it is. Have you ever switched it into desktop mode? I haven't pushed my Steam Deck as hard as my daily driver Linux system, but I've done all sorts of fun things on it like run a web server and write new Python scripts directly on the device. You can hook up a keyboard and mouse and monitor and use it like any other desktop Linux environment. It's basically just an Arch distro with KDE and some extra stuff on top to make it easy for people to run games.
I switched to Ubuntu after witnessing Windows 11 and am seeing there's now yet another confusing delivery channel (snap) added on top of what was already an overcomplicated system (apt). At least it still allows single installers (.deb files) so that works for now.
OSX (before the App Store model) had .dmg disk images and installer files, where you drop the entire application into a folder and that .app file contained everything the application needed to run. Easy enough to install applications.
But, what happened when you wanted to update a program or application? You went an manually downloaded the application or found the update button in the context menu and updated. This meant that many users would not update very often. Because the process was manual and because you were never sure what would work or when the new version would refuse to work with your hardware or OS version. Even with an app store, you still need to open the store app and click update or look through menus to find which software packages need updating.
For a long time, as both a Windows and OSX user, I saw the benefits of both approaches. Now, as a Linux user, I can update my entire system, with one bash alias I can update my chosen installed applications, but also my operating system files, and my flatpfak apps or containers and even my firmware (fwupd). It has changed the paradigm of computing for me. It makes me feel like a superhero. I don't have to worry about an update breaking my system. Better still, I don't have to manually update my operating system separately from each application I've chosen to install. I can do it all from the terminal and I can see all of the changes my system will make. It has been a great experience. Do things sometimes break? Yep. But, they broke on the other OS's too. I take it as a learning experience.
Maybe I've just talked myself into trying an immutable os.
1. You can have apps on both windows and Mac auto update from their respective stores.
2. Macs still use DMG and PKG installers, for stuff outside the App Store.
This is how macOS works with SIP, and how it handles rapid response updates for example.
It greatly reduces the ability for user space to compromise the system.
Which is an immense benefit.
> just steal all of your passwords
2FA, often now thanks to a HSM (Hardware Security Module) shielding your secrets precisely should your account be compromised (Yubikey, passkeys, ...)
> and credit cards
2FA. My credit cards companies (EU) ask me to sign on a physical hardware device the bank gave me any transaction I make with my credit card when it's either above a certain amount of to an unkown vendor (or both).
> and cryptocurrency
2FA. Cryptocurrencies hardware wallet use an HSM which shields the secret from attackers.
> and make user-level persistent processes.
Which you can detect from root, but only as long as root ain't compromised too.
A local exploit which can be detected and patched is bad but it's not anywhere near as bad as a root exploit which could potentially control the entire boot chain (maybe not SecureBoot if it's setup properly) and lie to you about everything.
Put it another way: it's precisely because a local exploit is not a root exploit that a system can be configured in such a way that should a local exploit happen, the system can make sure that that local exploit doesn't get to stay persistent.
A non root exploit cannot lie to root, which is why there's a distinction between a local exploit and a root one.
Now we begin to have the possibility to boot a minimal immutable Linux distro (maybe even from a read-only medium like a DVD [1]) , maybe from a UKI and a signature enforced by SecureBoot, and from that minimal immutable system, maybe launch something like a VM and/or containers (I prefer my containers to run inside VMs but YMMV).
For example we can begin to envision the following:
SecureBoot -> signed UKI -> Proxmox -> VM -> stateless containers
I am very excited that this now begins to be possible.
Don't you see any value in that?
I don't run an immutable distro yet but I already have throwaway user accounts, mounted on temporary and "noexec" mountpoints.
If you tell me: "Here's a system where it's guaranteed a malware can never ever manipulate boot state", I'll manage to find a way to build a system on top of that where local exploit cannot possibly persists.
Immutable distros are working towards that goal.
And I definitely see where the value is.
Why not just go back to the days when every process could access each others memory?
Let’s just let every process run as root too while we’re at it, right?
Heck, you can still crash a car, so why wear seatbelts or have airbags?
This is the problem with strawman arguments like yours. They’re not rooted in reason and extrapolate infinitely.
Imperfect safety is better than no safety. More safety is better than less safety.
It also completely ignores that you can have different approaches to security for different parts of any system. You don’t just have single silver bullets
yeah, it have lots of advantages, and that's why it was the default decades ago for everything. (windows, bsd, etc)
then people would have lots of trouble installing different software or updating security issues. so we invented package managers and took all the time in the world to make the base as small as possible.
it's advantages still make sense in some places, like modems with old flash memory. openwrt is static base with overlays. but again, it still carries the same downsides but because of the different aspects of the hardware it makes sense.
it would make sense for tech illiterate end users (hence android, ios, chrome os, wii, macos to a degree, etc) and containers (which already have infinite ways to convert from package to static). but anything else will literary harm the distro ability to evolve and adapt to software changes. imagine every change like systemd or a new browser or wm being atomic.
now people forgot decades of history. and it's so tiring.
When was Windows ever immutable in the sense of current immutable Linux distros? I wasn't able to find any reference to this ever being the case.
What do package managers and making the base as small as possible have to do with immutable distros? Package managers still exist, and the base is pretty much the same size as the non-immutable version of the same distro.
Why do immutable distros make more sense on modems with old flash memory?
How does being immutable harm the distros ability to evolve?
Either I'm not understanding your position at all, or you have a very different understanding of "immutable" than I do (after using Kinoite as my daily driver for a year).
Comparing nixpkgs to the AUR seems to reflect the opposite trend. Arch is hamstrung by a dependence on outdated and poorly maintained repos that cannot coexist with up-to-date packages. Unless you fully embrace static linking or atomic installs, you'll end up with breakage.
MacOS went the static linking route, and Windows wrote back-compat for most old software by hand. The "decades of history" hasn't proven any approach right or wrong. They're all flawed.
On an unrelated note, I despise the constant insinuation that using Linux is an indicator of intelligence whereas users on other systems are tech illiterate.
Actually, I believe your entire argument contradicts itself multiple times because you give examples on both sides, and don’t stop to reconcile your views.
During boot time, a directory was checked for update packages, and if the public key signature of the package matched, the updates would be applied before the filesystem went into immutable mode. This update directory was one of the few mutable directories on the system.
Years later, I gave a daughter of a friend of my mother my old PC. It would boot up a Linux live CD. That, too, is immutable, and you'd update it by burning a new live CD.
But where did we arrive to this? Well, computers had all services enabled for some reason (not with big bad internet in mind, but LAN). And updates were distributed via CDs or different media. Some airgapped environments are still going to work akin to that. Now, if the devices are connected to internet, they have to be updated because security vulnerabilities are going to have been discovered.
Someone mentioned running Puppy Linux from a CD/DVD (write once).
I do wonder: it'd probably be possible for me to boot a Linux distro from a DVD and then launch Promox and my VMs/containers automatically. I take it I'd have to burn a new DVD every time a security patch affecting programs installed on the bare system comes out.
The "main" OS would be hard to compromise in a persistent way as you cannot remount a write-only DVD read-write.
I’m not quite sure what’s going on here yet, but seems interesting
$ apt search btrfs apt
Sorting... Done
Full Text Search... Done
apt-btrfs-snapshot/noble,noble 3.5.7 all
Automatically create snapshot on apt operationsA cute name and a university student somewhere does not constitute stability, no matter how good the intentions. It's not a bad thing, but you have to know what you get yourself into. Most of the distributions listed in the article belong to the latter category.
Immutable systems are great for embedded, network equipment, appliances and industrial applications, and specialized distributions for those applications have largely been immutable for a long time already. Nobody really wants an immutable system for their main desktop, because working is all about mutating state. You write may write documents, save bookmarks, install plugins, or try new software. Those are the things really immutable systems like kiosks wants to disallow.
So in order to make for usable software these desktops generally split your system into a mutable user part and an immutable system part. That's basically how unix-like desktops have worked since forever. Stuff in /bin and /sbin is only changed by the package manager. So the fit is quite good, but it also means it really isn't as useful as it's made out to be. That's why most people don't use them.
The use case is mostly for rolling back updates, not really running from readonly filesystems or preventing change in other ways, but most distributions already do that. You can roll back updates with both dnf and apt. It's not perfect and doesn't always work, but mainly from a lack of testing. With snapshots it's pretty much infallible though.
My recommendation if you really want something that "just works" is to install one of the major and time tested distributions. Pick Debian if you don't know what to choose. And then learn how to use it. Anything these tiny experimental distributions offer, such as running off read only filesystems or rebuilding it for your brand of cpu, or testing a new desktop environment, is likely possible in Debian too. With the added benefit of it being around in 20 years. And the core distribution is less likely to break in some way because some maintainer found inspiration for something. As long as you don't run untrusted stuff as root, stay out of the system files, and generally let the package manager do its job, you're going to be fine.
What I would like to see a desktop distribution work on is basically the same things as 20 years ago which still isn't really done outside some exploratory work (probably because it's actually hard):
- Packages on a user level where it is easy to install new stuff without touching the system area. More tricky in practice than in theory because of state changes to configuration files, saved file formats etc. But some should be easier than others.
- Desktop software service accounts, just like we do for server software. Mostly relevant for larger packages such as Firefox, Libre Office, movie players.
- Integration with popular third party package managers from the language ecosystems. Most language packages are anemic. All the powers that a package manager gives, reporting, listing untracked files, listing changes, rolling back updates, should be available for them by integrating directly with them. Package definitions should be able to be imported without manual work.
- Package managers should have at least some knowledge of an application's access patterns to help with application confinement. Still today things like selinux policies are packaged are separate entities and managed with external tools, which brings a lot of complexity since all possible configurations must be supported there. A package manager knows more about the system and could handle these files. Confining desktop software is a usability problem more than a technical one, but it is clear that desktop environments needs something to build on to make it practical.
RHEL peeks from behind some furniture
We all believe foolish things at some point in our life...