Switching from Arch to NixOS
rugu.dev
rugu.dev
As the sibling comment describes, Nix can get a bit gnarly once you're off the beaten path, and having your whole OS subject to that is a pain.
But as an alternative approach I run Fedora, and install key stuff through the OS using dnf in the normal way, but then I have a home-manager config that covers all my key config, and a lot of the extra applications.
Yes, I don't get the full "value" of a whole system under version control. But I have a very stable, well proven system with few surprises. And I also have good version control of my config and day to day apps.
Plus the extra Nix stuff - if I want to try out a program, or use it temporarily I can just launch a shell with it and it's not cluttering up my system for ever (e.g. when exa was on HN yesterday, I could just do `nix-shell -p exa` to get a shell with exa in it. At some point in the future it'll get garbage collected again).
As the story goes, Ubuntu had just released 24.04 and I pushed to upgrade. My linux partition was on btrfs. I had been bitten by the snapshots-locking-my-system problem before so I cleared out all my snapshots (and backed up my drive to local NAS). During the update Ubuntu did ANOTHER SNAPSHOT, it took up all the space on device (256GB, yeah I should have had more space) and crashed the system. At that point I was finally done with all of the shenanigans that Ubuntu pulls with apt (not updating everything unless you have their subscription, forcing some installs to go through snapd, separate driver management from apt…), and this last update crashing my system’s boot, even though I could have recovered with my backup, was the last straw.
I considered my experience with Ubuntu, and MSFT’s actions to integrate Copilot into the OS, and decided to wipe the entire 1TB drive and go all in with NixOS. I was able to move my home-manager config right over, get my configs all converted to flakes, and get hyprland set up all in less than an hour. Smoothest transition in a long time. I even was able to jump to the 6.8 kernel and things just started working again. I was finally able to play steam games well on my system again. I may even have a path to get my VR to work in Linux again.
Are there still problems? Sure. Hyprland is a compromise because sway still won’t cooperate with Nvidia GPUs. But, when I have made changes that get me to a non-bootable state, I can boot to a previous generation directly from GRUB, see what went wrong, fix it, and get back to a good state. Having that ability to recover from a bad boot situation without hacking into a safe mode or GRUB console or rip the drive out and put it into another system was incredibly refreshing. I am not going back anytime soon.
This Fedora installation is 5 years old now. I have done a distro upgrade at least every year. It has worked extremely stabley and predictably all the way though, including the Fedora transition from X to Wayland.
I did not like my own Ubuntu experience when I was using few years before as well. I don't think its stable enough for newcomers as well.
For me, years and years ago, that 'pain' (having my whole 9S subject to Nix's constraints) is what pushed me to really learn it instead of just retreating to the traditional base system. It transformed learning Nix (and even sometimes writing packages or extending NixOS) into the path of least resistance to run the software I wanted or needed to use. And that was helpful for me.
For others (and for me at other times in my life) I understand that this is not always the right approach. But it can be really good.
Well, yes actually, you can get help in IRC/Matrix/etc. It's best effort and might not help, but it's not like commercial vendors promise fixes either.
In any event, Arch is in the same position so that's not really an argument in favor of either one.
I'm in my quest to find the "best supported" way to run ZFS on Linux and I've tried Ubuntu, Arch, RHEL, OpenSUSE. (I also tried FreeNAS (i.e. FreeBSD) in the past.)
Ubuntu has knowingly made a version of OpenZFS with (with their own patch causing) data corruption bug in the stable release. So since then I lost trust in them.
Arch Linux never officially supports it, and Arch's rolling release model with OpenZFS as kernel modules are not a good match for UX.
RHEL with kABI tracking kmod maintained by upstream seems to be a great, robust support combination, but later I found out that the kABI is not fully covered so it working relies on a bit of luck. AFAIK, it is still not completely resolved.
OpenSUSE with the "filesystem" project seems to be a semi-official support of seamless integration of OpenZFS with OpenSUSE. So far so good but I'm still evaluating.
In principle, nix and nixOS should be quite robust from its design (practically reproducible build). But I am not trusting nix that much yet and prefer something more battle tested. Coincidentally, I like the documentation of all the previous OSes I mentioned above (where Ubuntu falls a bit shorter among the crowd.) And from my limited experience, nix and friends are quite poorly documented.
My main laptop, and the main laptop before it, have run NixOS with ZFS root since November 2022. Commit history seems to show that I started on version 22.11 and I think I've stepped forward through every version since then (might have skipped one in the middle, not 100% sure; I didn't switch to flakes until partway through and channels aren't in the commit history). In that time, the only problem I can recall ever hitting is accidentally filling up the /boot partition because I wasn't cleaning up old generations. If I was doing it over I would either make /boot like 8GB, or just use zfsbootmenu so the kernel+initramfs live in the root filesystem and it's irrelevant. Other than that, it's been smooth sailing.
> I'm in my quest to find the "best supported" way to run ZFS on Linux and I've tried Ubuntu, Arch, RHEL, OpenSUSE. (I also tried FreeNAS (i.e. FreeBSD) in the past.)
Ah yeah, if the rest of your software+hardware+preferences support it, I would easily believe that FreeBSD is better at ZFS than any Linux distro. IIRC they even ship boot environments in the base system which is great.
> Arch Linux never officially supports it, and Arch's rolling release model with OpenZFS as kernel modules are not a good match for UX.
IIRC there's some unofficial repo that provides both the ZFS packages and kernels together? In any event, in NixOS just use
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
> In principle, nix and nixOS should be quite robust from its design (practically reproducible build). But I am not trusting nix that much yet and prefer something more battle tested.
One advantage of ZFS is that you can compliment nix's versioning (seriously I do recommend using flakes and keeping your system flake in version control) with regular old ZFS snapshots. Just run `zfs snapshot -r rpool@todaysdate` before any change and you can roll back just about anything. Also you can zfs send those snapshots to some external storage and have full system backups with easy incrementals as desired.
> Coincidentally, I like the documentation of all the previous OSes I mentioned above (where Ubuntu falls a bit shorter among the crowd.) And from my limited experience, nix and friends are quite poorly documented.
Yeah, that's fair. It's improving with time, but anything off the beaten path tends to devolve into random blog posts. OTOH, if you keep the NixOS system to a simple base system and run most of your stuff in distrobox or some other container, you can have the best of both worlds.
The thing I worried most is not the kind of problems fixable by rolling back, but (potentially silent) data corruption. Do you know how exactly is OpenZFS on nixOS supported? For example, currently both the Arch Linux and nixOS guide in the OpenZFS documentation said people should mention @ne9z in filing issues on those platforms, but @ne9z no longer exists.
I have seen how maintainers can screw up and causes data corruption (Ubuntu specifically) so I worry about that aspect as well.
This largely reflects my experience with NixOS as well. I installed it for a few months on my laptop (on two separate occasions) and there are parts of the OS I love. Being able to simply define your packages and user setup in a single configuration file is incredible, and even now if I wanted to switch back to Nix, I could dig up my configuration.nix and pick up where I left off. Gone too are the piles of mysterious packages installed on my system but I can't recall why they're there or if I still need them.
Unfortunately, once you start stepping off the beaten path, Nix often became a source of frustration. If there's a well working package available in its repositories, great. If not I found creating my own Nix packages to be an experience of arcane incantations, even for the simplest things. Compare this to writing Arch AUR PKGBUILDs, which are simple, glorified shell scripts, and it quickly led to me growing frustrated with the OS.
I think what eventually did me in was trying to get Neovim LSPs installed correctly. For some reason I never really understood, where ever my Nvim plugin was installing them was certainly not where Nix wanted them, and I never resolved the issue. Things like this eventually drove me back to the comforts of Arch, although every once in a while I look back over and wonder what could've been. I think it's the only distro I've tried in recent years that really tempted me to switch permanently.
The ideas of NixOS are fascinating, and I really think it can create a new paradigm in the Linux space. However, I'm unsure whether it'll be Nix itself that really gets there.
My experiences with NixOS almost exactly mirror yours. I like the concept to a degree but find the implementation flawed. If the project was more cohesive and unified (i.e. OpenBSD); then all the edge cases wouldn't combine to make it less then the sum of it's parts.
I love the idea that every configuration could be treated like a cooking recipe. Here's a list of all the ingredients, this is how I want them combined; and poof it's 100% repeatable.With that being said, I’d love it if a NixOS like variant came to a BSD.
https://github.com/nixos-bsd/nixbsd exists, though I've not tried it
For servers that's even more relevant: I usually logged into my home raspberry pi every once in a while because I needed to fix something and I didn't even know what services were running on it!
With NixOS it's all comparatively simple: I switched my gaming PC to NixOS a few months ago and while I had to reconfigure some things like display settings, everything else just immediately worked exactly the same as on my laptop after installing.
I set up nix-darwin on my work macbook and share all my dev environment's configuration with my other computers. Now if I change some Helix settings on the macbook, I just have to commit them to my flake (which is just a git repo) and to get those changes on my desktop as well I just have to pull and rebuild.
If I want to install some new service on my home server I just open the relevant configuration.nix file in my config, enable an option, build it locally to check that everything's working, and push the changes to my server.
I don't ever feel the need to do a clean reinstall of my system because everything is already as clean as can be, and if I really wanted to it'd just be a matter of backing up /var and /home, putting the nixos installer on a usb stick, and running nixos-rebuild from there.
I think the benefits of NixOS come into play months down the line, when a problem comes up and all the specific knowledge about what you did with your setup has faded from memory, but all the operational knowledge on how to use Nix is still there.
Edit: I'd recommend trying out NixOS a server or VPS and installing Nix with home-manager on your usual distro for a while. That might give you a taste of what you can do with NixOS without having to fully commit to having it as your main OS and doing everything the "nix way".
This is accomplished via the shell command:
'$ nix-shell -p <name of software>'"
i.e.,
$ nix-shell -p vim
Where <name of software> can be found from over 100,000 packages (here):
Functionally, the above is the equivalent of what Google Play Store, Apple App Store or Microsoft Store or what other Linux package managers are to their respective platforms...
To install software permanently, edit /etc/nixos/configuration.nix to include the name from above (typically under environment.systemPackages), and follow up with:
$ sudo nixos-rebuild switch
Also, it's not too difficult to undo those changes should that be wished, but that's beyond the scope of a simple HN post...
Anyway, in my opinion, NixOS is probably the most Software Developer friendly evolution of Unix since Linux itself.
Well worth a try for technical people -- but there is a learning curve when trying to do very specific esoteric things with it(!)... but that's sort of the case with anything technological...
That being said, there's lot more documentation about NixOS on the web these days than there was in the early days... so it may be worth a look!
But I don't know what the current state of it is as of now
Yes this is the start of a very steep learning curve but I have been taking it very slow and it has been working for me since day 1. I think it’s pretty brilliant that the entry level is so low.
Experimentation is cheap (just use grub to boot into your previous env if you mess up, try packages with nix-shell), and I have had great help in the Jupiter Broadcasting Nixnerds Matrix channel. Some great people are in there.
I've switched to using devenv[1] as a first step. This at least allows me to experiment with using nix to configure my dev environment in a per directory fashion, while allowing me to use the Linux distro of my choice.
Sounds like having NPM as an OS. Absolutely no thanks.
I'll stick with "difficult" package managers that seem to resolve things simply rather than having 27 different copies of the same thing and having to worry about things I didn't before. If, for whatever reason, I need two disparate versions of something I'll figure it out myself rather than abstracting it away.
In my mind, this move seems to be moving from 'high potential energy' to 'low potential energy' where you don't have any room to escape; you have locked yourself at the bottom of a pit and you just have to hope things don't roll over the top because you are at the whims of what NixOS has decided. If you were in a 'high potential energy' situation (ie: Basically any other distro), you would have room to fix it yourself or even take your widely compatible dotfiles and move to a different OS. Even if you are allowed to do configuration outside of NixOS as written, I am going to guess that NixOS isn't a huge fan of it and that configuration is fragile.
> However, after using NixOS for a while, I realized this was not true. The abstraction that NixOS packages provide doesn’t hide everything from the user to avoid confusion with irrelevant details. Instead, it offers a way to configure your environment the Nix way, so the resulting configuration files are created by Nix.
I'd rather beat my head against a wall for a day figuring out how to configure something than get locked into an OS-specific configuration that I can't move elsewhere.
Also when you combine it with this:
> The documentation is not in great shape.
Absolutely no thanks. You are telling me I have to do things the "NixOS way" but also your documentation is bad / inferior? Absolutely no thanks.
That's not how Nixpkgs works. Nixpkgs, the collection of packages that NixOS is built on top of, is a monorepo with only one version of each end user package. There are only multiple versions of a package in a given commit of Nixpkgs if that is necessary to support some other important tuple of packages because they each have a hard dependency on different versions of some package— just like on your typical Linux distro.
The ability to have multiple versions of something installed side-by-side comes into play for your own voluntary customizations, and (optionally) for some third-party package sources.
Nix doesn't feature other NPM insanity either— there's no network access at bulld time, no circular dependencies, etc.
> I'd rather beat my head against a wall for a day figuring out how to configure something than get locked into an OS-specific configuration that I can't move elsewhere.
Many people use Nix and home-manager on their MacOS or non-NixOS Linux setups. You can use Nix package manager without committing yourself to the constraints of NixOS.
> You are telling me I have to do things the "NixOS way" but also your documentation is bad / inferior? Absolutely no thanks.
Don't know about others but I don't tell you that for sure. I think poor documentation is among the most understandable reasons to NOT use Nix.
In the end, I am just trying out things to see whether they benefit me or not. Not here to sell you anything.