NixOS and my descent into insanity
ersei.net
ersei.net
But, man, it's so slick when you get it working. It's so easy to understand your system and how it's set up. There's no weird stuff changing between updates that you don't know about. You don't have to merge new configurations. It's so easy to try something new without changing your system. Rollbacks are awesome. It's easy to move your config to a new machine or keep multiple machines in sync. My machines were constantly getting out of sync as far as packages, versions, config, etc. and it was a headache to maintain them all the way I like. Not anymore. It's so easy to maintain working NixOS systems. I have the same packages, users, and config on my machines without having to think about it. I have automatic updates going where I get a new system when I reboot. It basically works like an appliance I don't have to think about except it's set up the way I want it instead of how someone else decided it should work.
There are so many nice things about NixOS, but it sucks. It's the worst thing out there except for everything else. I hate it and I feel like I can't go back either. I would love a NixOS-like system with a better language, documentation on par with Arch, and the reliability of Debian.
Another language won't change much.
Debian reliability is outside of core packages a myth and Debian has no way to properly migrate configuration. Every major update is a wild rollercoaster ride were things might break in the middle.
Why do you think Guix+Guile isnt a substantial positive change?
The language is very small and is a great fit for its domain.
I personally moved from UBlue to NixOS because I was a bit bored of running my whole life in Distrobox, and didn’t like being so dependent on GitHub. NixOS gives me everything I liked from Silverblue with the flexibility of something like Arch. Plus, Home Manager is amazing. The big downside of container-based images is that you can never touch your home directory. Now instead of Silverblue plus Chezmoi plus like 7 Bash scripts and a Distrobox, it’s all just one Nix config.
But I do think we should keep in mind what an incredible project NixOS is. NixOS isn’t just some other Linux distro, repeating pretty much what every distro has done for 40 years. It’s a complete rethink of how a Linux distribution and package manager should work, from first principles, in a way that I don’t think has ever been done before. That’s an incredibly bold thing to do. That alone, as a research project, would be an impressive achievement. But it hasn’t just ended up as some researcher’s PhD thesis. It works in the real world, and it’s built a large community despite its learning curve.
So despite the terrible docs and UX, I have trouble agreeing that it “sucks”!
I get that backwards compatibility is really important, but still. Gotta be POSIX compatible at least or else next to no one is going to use it as a daily driver.
For example:
> Coming up with a custom shell language was fun, but ultimately does not solve any real problems for us.
> Having a POSIX compliant shell would allow us to, well, run POSIX shell scripts. (This currently requires installing a 3rd party shell.)
https://github.com/SerenityOS/serenity/issues/6847
Where's some variety? I hope to get into the field myself eventually, so maybe I should take it upon myself to create interesting things that might be useful.
Not just Linux but the OS in general.
As far as i can tell no one (thus far) has been able to explain a reasonable and concrete way to tell what any given variable is within a Nix source file. `nix repl` works decently for some cases, but many not - since it seems to mostly help you inspect the output of a Nix expression. So hypothetically it could work, but for any given variable you'd have to recreate the entire value/function/etc stack that produced that value in that state.
All around after ~16 months i think i'm finally done with Nix. It's Nickel or the highway for me. Not that i even love Nickel that much, but it's the bare bones of what Nix should have been in my view.
Not sure if it's a hot take or not.. but it feels like you can have Lazy or Dynamic, but never both. The DX is terrible with both.
That is, I ought to be able to do something along the lines of
`let x = break (...)`
The semantics here is that, when x is evaluated (i.e., forced into WHNF), I get placed into a debugger where I can examine the context and either (1) choose to return a new value for `x` or (2) let it fall through and let `x` take on the value of `(...)`. The main issue with nix is debuggability
Nix is just a massive blob of difficult to debug code. Really awful experience for me.
[0]: https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...
For example, i don't see why a configuration would install actual packages - if that's what you mean. Besides, i am not advocating against lazy by itself. I'm just saying that something which is both lazy and untyped is a huge PITA. Nickel adds typing, validation, etc - information, by which you can still have Lazy and a better DX by way of types. At least in theory, i've not used it much.
My biggest gripe is dynamic Nix, fwiw.
Nix is lazy because the whole point of the original work was "what if package management, but FP". The resulting /nix/store etc system has very little to do with FP and is, in fact, built by imperative bash scripts.
That’s the problem with the anemic and ad hoc libraries nixpkgs uses, not the language. There is a magic buildWhateverXPackage function that needs some magic attributes and you are done. The problem is that there is no interface to aid in how that should be used - but it is the same way in JavaScript as well.
Also, you are writing a description of something, it won’t be a traditional program with an event loop or whatnot - here lazyness does make sense.
It's certainly not ergonomic, but from what I can tell so far, `builtins.trace` and `builtins.traceVerbose` are about as good as it's gonna get. It quite literally lets you inspect any arbitrary Nix expression, but it's just such a friggin' hassle to wrap things with it & reevaluate just to do what literally any real debugger can do in two keystrokes.
> All around after ~16 months i think i'm finally done with Nix. It's Nickel or the highway for me. Not that i even love Nickel that much, but it's the bare bones of what Nix should have been in my view.
I'm curious. Have you messed around with Dhall? It's suspiciously similar to Nickel (even mentioned on their site), but it's been around longer so it's probably a bit more viable to actually make your config in Dhall. Dhall or Nickel, it'll still insanely janky and more trouble than it's worth. Personally, I've found that what's worked surprisingly well for me is just really focusing on building a good understanding of the Nix language in an effort to avoid needing to debug as much as possible. "Just don't write bugs" as the saying goes. I've been using Nix for a while, and I'm still loath to debug it.
edit: ooo, looks like someone made a similar program with Dhall. https://www.haskellforall.com/2017/01/typed-nix-programming-...
I didn't use it yet, but looks like this supposed to do it.
I'm now exploring transpiling to Nix lol. I'm moderately dedicated in my distaste of Nix hah.
This is what confused me the most.
Nix (CLI tools / language syntax)
NixOS (A Linux distro that has CLI tools, but also some CLI tools that are only available on this distro and not others, aka "nix" doesn't really work on Debian)
Flakes vs "configuration"
I just wanted to like "terraform apply" a system configuration on my Debian install.
People are obsessed with Nix because you can manage your config for their local system
Ok... Ansible has existed for 12 years? And you can use it on Debian/don't need a full OS/distro from the ground up (NixOS)
But people will come out of the woodworks to tell you how you don't understand, you're wrong, you're missing the point, Nix is the greatest.
- Language agnostic software packaging
- Package manager
- Ephemeral dev environments
- A purely functional config language (it has lambdas - you can build abstractions)
- Closures - Nix knows every single dependency your system needs down to git revisions
- Binary caching. Everything is built from source, but you can safely download binaries that are known to be built from the exact same deps you need.
- Ergonomic cross-compilation
- NixOS modules allow for composable, modular system config. For instance, my home computers all import a shared scanner.nix file that sets up scanner drivers and installs scanner software. Sharing config across machines is so easy with Nix.
- Rollbacks - you can edit your system (even swapping out gfx drivers or the kernel) and if it doesn't work, trivially rollback. As in you can restart your computer and choose to boot into the previous config.
- Easily modify _any_ dependency. You can apply patches, point at a fork, or just tweak its packaging.
- I am surely forgetting stuff! It does a lot.
Ansible/Chef/Puppet/Saltstack
They try to be declarative (following less popular CFEngine) but they are still iterative. Saltstack is closest as originally the order of operation was based on the declared dependencies.
They still ultimately come down to just perform iterative steps, with the difference that operations are idempotent.
The biggest drawback they do have is coming from the fact that they have system with a state and they modify the system state to get selected files/services/packages/etc in new state.
Nix lives up to its name, and when configuration happens it starts from nothing and then builds everything based on the configuration.
The result of it is, let say you use the traditional CM tool to install a package. Then later on you decide the package is no longer needed so you remove "pkg.installed" declaration. What actually happens though is that the existing systems will continue to have the package installed (until you add "pkg.removed"). What's even worse is that any new instance you spin up will not have the package installed, so everything will be drifting. This is the primary reason why there was the immutable system movement. That's all to avoid those scenarios.
In contrast, in Nix, if you remove package from list of installed packages, or you remove file that you declared by nix config, they will be gone from the system. The configuration is basically describing your system.
Now, as mentioned above, that's just one piece of things Nix can do. I primarily use it for reproducible builds I really like that if a package won't build on your computer it probably will fail to build on mine as well (it's something that docker promised, but IMO failed to deliver), but let's stick to configuration management.
Another big thing is that all changes are atomic, you either get your change or the change did not apply due to error, you will not get in a in-between state, like frequently happens with CM.
Another thing is (and what I wrote above is prerequisite for this to work correctly) is that you can do rollbacks. And it works with things that are considered hard. For example NixOS by default use X11, but you can enable Wayland with some configuration changes. Then you can just revert the config back and get back to your old X11 installation.
This also makes system upgrades less dreadful.
Let's say you have your setup, but you need to use different version of openssl or maybe compile it with different flags. In traditional OS you probably would not do that or if there was a vulnerability you would wait for help from vendor as there might need to be multiple packages updated. But if it is something smaller, maybe you need python package, you would build your own version, you would place it in artifactory, then have ansible task to upgrade it on your instances.
With nix you will use overrides[1]. Nix then will see it, and if the compiled version is not in its cache it will pull all dependencies and recompile it automatically then place it in its cache.
If you set up a binary cache yourself (you can either use SaaS solution, or even just set up an S3 bucket) you can have your machines automatically pull the compiled version from it. Nix doesn't care about versions of packages either (the versions are only for the user) instead it uses cryptographic hash generated from things like source code, package dependencies, architecture, compile options, etc) this means if you cache compiled version and modify package to add some other options, it won't accidentally use the other package it will either pull the right one or recompile the code.
While these tools are good at making certain changes, and are even pretty good at detecting and eliding already-done work, they are _not_ idempotent. There are nearly no modules that properly handle unmaking their changes, so once you mutate that bag of state you're left with a state vector that is barely better than completely undefined.
If you're familiar with these tools you have no doubt deployed undo-this-change tasks that you leave in the tasklist until you're fairly confident it's run on every system the do-this-change tasks ran on. Even if a module does attempt to offer an undo-the-state functionality, you can't just remove it, you have to first configure it to run in the "undo" configuration before you remove it.
I am also mostly an Ansible-on-Debian user, and know for a fact that NixOS has a few things that might be interesting to me in the future. But I don't want to be yet another person "coming out of the woodwork" :) But for a lot of people, Ansible-on-Debian should solve their problems pretty well!
I think 'like Terraform, but for the system configuration' is an apt description of NixOS. -- i.e. NixOS is a Linux distribution built around using Nix to generate the system config.
The Nix package manager itself happily works on other Linux distributions. (And on macOS).
> ... Ansible ...
Nix's main focus is on packages. It happens to incidentally do package-related stuff quite well, which is why nix also gets used to declare development environments, build Docker/VM images, or used to configure NixOS.
An interesting set of terms related to Ansible vs NixOS: Ansible is 'congruent', in that it tries to get a system to a desired state by comparing what's there with what it expects. Whereas, NixOS is 'congruent', in that the system is built to be in the desired state.
https://flyingcircus.io/blog/thoughts-on-systems-management-...
But not the "Nix system config manager", which is grouped under the same set of CLI tools, is that right?
> Nix's main focus is on packages.
Packages or system configuration or both?
The `nixos-rebuild` command is what you'd use to apply a NixOS configuration (that's NixOS only). (I believe it's possible to run the command to build a configuration on non-NixOS).
Whereas, the other nix CLI programs are mainly `nix-build`, `nix-shell`, etc. which are now grouped under a single `nix` command.
> Packages or system configuration or both?
I'd say packages.
That people like NixOS for its declarative system configuration is incidental.
Nix's key idea is symlinking everything. You put a package in some directory under /nix/store, where the address is computed from that package's inputs. (e.g. what compiler to use, what compiler flags to use).
If you build the same package with different inputs (e.g. different compiler versions), these packages can exist alongside each other in /nix/store. Nix can then build a PATH using whichever package you want to use. It's able to come up with ~/.nix-profile/bin and ~/.nix-profile/include (and share, etc.), symlinking to the stuff in /nix/store.
NixOS essentially 'just' applies that idea, but symlinks stuff under /etc to whatever /nix/store package has the system configuration. (Plus uses systemd services, and a few other things).
e.g. when I `readlink /etc/hosts`, it resolves to `/nix/store/n5ijm9zzkcm7qglsqrfvnixih0q7dskb-host`, or /etc/fstab to /nix/store/xb7jmzd0wy0wy4vihh0q17m81kmqgc7y-etc-fstab.
Except Qubes OS that allows using original GNU/Linux distributions virtualized with a convenient interface.
I am curious, what is inconvenient in it for you?
That’s the same idea that Docker has — containers don’t solve packaging, they only push the problem down a layer.
I run Fedora and Debian, but you can run many other distros: https://www.qubes-os.org/doc/#operating-system-guides.
> they only push the problem down a layer
This is an interesting interpretation. For me Qubes fully solves, e.g., both the problems of system stability and flexibility.
But at the end of the day, those VM images, and any snapshots made, are opaque blobs of bits. Whatever shape they're in, you have nowhere to look to determine why they are the way they are--no recipe for recreating that state and sharing it with others (unless you want to share it ALL with others, and then only with others on the same architecture). You've got a lot of control, but only over these chunks of whatever happened to be there at that time.
You're sort of trapped in one dimension. You can use snapshots to move backwards and forwards in time, but if the following paths don't lead to the same system state in some subtle way you're not going to know about the difference:
path one:
> add foo | add bar | remove foo
path two:
> add bar
Did you consider to use the Salt Stack, which is integrated into Qubes? https://forum.qubes-os.org/t/saltstack-challenges-and-experi...
Do Qubes users treat Salt Stack formulae as packages these days?
I grok the first two, but what do you mean about the reliability of Debian?
I used to love Linux, about 10 years ago, but it seems to be stagnating.
Stuff basically never breaks on Android, so it's clear you can totally make a computer that actually works. But even mint occasionally has random fussy problems, and so much of the internals are built around manual admin.
I mean, debian packages can randomly decide they need to ask a question during setup!
Plus Nix seems to have modern packages. Other linuxes have a huge repo, but a lot of it is nonsense, unmaintained, or obsolete, or something only of interest to unix fans. With nix ever fairly obscure stuff that normally needs a ppa seems to be there.
Home manager is a well-intentioned and technically well-executed piece of software that is nonetheless a disaster as the “how to do dots on Nix” story, especially for newcomers. We need “Home Manager Light” or something.
I’m a fucking serious Nix-head and I procrastinate doing anything serious to my dots like getting a root canal.
This is a critical use case for the apprehensive crowd like me who is averse to distro hopping, let alone willing to take on the paradigm shift of Linux-to-nixos.
The documentation for either is all but non-existent and the pages I can find are glorified source or man pages. Given how scalable nix, in the right use case, can make things, that's just not the modern experience it should or could be.
I suppose, I can't have the second part without NixOS, though ...
> If the profile doesn’t exist, it will be created automatically.
Nope! If it doesn't exist then running the commands this tells you to will bork your installation completely, and unrecoverably with the knowledge you have gained up until that point in the book, at least without starting from scratch again.
Search for issue - closed in 2018, wontfix as "It's documented correctly and quite clearly." by someone who apparently didn't read or understand that the literal reference manual tells people to break their installation.
I have never touched Nix/NixOS since, because clearly they don't want or need people without 100% of their lives to dedicate themselves to it.
Tip I use for this:
End your ~/.ssh/config with Include ~/.ssh/config.local . Write ~/.ssh/config with whatever you're happy being public. Write your aliases into ~/.ssh/config.local. SSH will just ignore it if it's missing.
This has also been my experience with home-manager. The promise is great - it's like DLC for Nix that just Does More™. Then you get into it, and realize the modules are a lot more fragile and most of them are outdated. Then you're a few hundred lines in with workarounds, DBUS declarations and all sorts of declarative ricing and go "is it worth it?".
Since then I just removed all the home-manager stuff and relied on NixOS' native config. Things are much more stable and easier to define, and I haven't found a whole lot of things that are truly undoable without it. It's slower and non-declarative, sure, but my sanity is all the better for it. I hope I'm not the only one out there.
Their technology is on point.
The development experience of flakes is just unreal. Having seen it in-use at work was eye opening:
- You get all the Nix goodness, including package/dependency caching for multiple OS/arches with minimal configuration. Build times are practically reduced to automated searching and downloading.
- Dev environments are much cleaner and not spread across READMEs and makefiles. Adding a build dependency is one (1) SLOC in flake.nix. Adding environment variables is trivial, and it all gets shared with your branch.
- Invoking a quick 'nix develop' drops you into a shell w/ build dependencies, `nix build` builds the program, `nix profile install` adds it to your current environment, etc.
The list goes on. It will clean up shop once dev teams find out how little configuration is required to get this all working, if Nix move fast enough to ship this.
Flake is a worthwhile addition to Nix that is worth learning. But like anything Nixian, it's not straightforward.
Have you checked out any of the tools that aim to simplify Nix experience? We built Devbox (https://github.com/jetpack-io/devbox) with this in mind.
Terrible UX. Terrible documentation.
Is Guix any better?
I installed it on Debian, and there's no uninstall. Sure, at that point I was already frustrated, but IMHO that's just a sample. It wasn't for me.
Those constraints don't really apply if you have admin control of your machine.
(And the troubles in the first half were caused by zealously diving deeper with Nix, after an initial simple setup went well).
The excellent Arch Linux Wiki demonstrates that high-quality community-maintained docs are possible.
In reality Nix is a programming language. It is a relatively simple language, but it is also functional and lazily evalated. And while the language is simple, its stdlib (nixpkgs) is actually puts another abstraction layer on top of it which is quite complex.
I do think though that Nix should do the same thing other programming languages do: identify what the nix stdlib is, move it to a separate repo with its own release schedule and ensure that whatever is added to it/removed from it is documented.
This won't help author of this blog, but it would be a great help for anyone who is willing to go through Nix Pills.
Personally I loathe DSLs unless you genuinely need something really fast and really simple. Otherwise please just pick something off the shelf and save everyone the headache.
The Nix language is functional and lazily evaluated, it closely matches how Nix works builds packages, which basically makes it less verbose.
Even if they used python (or python like language like starlark), and then proceed to build an abstractions on top of it, barely documenting the function. Leaving old ones with the new ones, you would still be confused and frustrated needing to look at the source code.
Also few things:
- he decided to manage sway via home-manager, then got suprprised that after he made changes to its configuration and run "home-manager switch" it restarted the service
- when searching for installing nix on another machine, he tried nix-user-chroot, nix-portable, proot. He pointed out that nix-user-chroot is abandoned. The reason is that it is no longer needed, what he wanted to do can be done with statically built nix[1]
[1] https://youtu.be/6Le0IbPRzOE?t=4282 - I recommend watching the whole video (especially once he starts doing demo at 14 minutes) as you can do a lot of things you might not even know it was possible.
I still get frustrated that some new features that are being added are undocumented, though I also noticed that the nixpkgs documentation by default is set to last stable version. This means changes could be delayed by even 6 months. And given how rapidly nixpkgs change that could be the reason.
FWIW, I see that nix-portable does support nix flakes, and home-manager also supports nix flakes. (And flakes are a feature which replaces use of channels).
Sorry, what? If your zsh is slow in a big git repository, that's on you, not zsh. I honestly have a hard time taking anything else in this article seriously after reading that.
edit: the article is good. I was being harsh. Nix is horrible, and yet I refuse to use anything else.
> I was getting kinda sick of [my Zsh] being slow in big Git repositories. Instead of fixing [my Zsh], I decided to give Fish a shot.
Yeah, it can be hard to get something working, but once you do, you can be sure it will work anywhere.
Until I hear differently, I'm just gonna stick with my battle-tested shell scripts and symlinks. Sure, it's not the same degree of reproducibility, but also I've not yet had a practical reason to need whatever remaining percentage of the diminishing returns Nix offers that I'm not getting, at least when using my scripts combined with the use of other tools such as venv, nvm, cargo, etc, or even just building a lot of software from source.
The fish-related issues came from trying to manage the whole of the fish configuration with Nix.
I think that's a fairly niche thing to want to do.
I'd say NixOS's declarative config more/less covers the config files, where on non-NixOS you'd edit the files under /etc (and hope you didn't break anything).
Whereas, Home Manager's declarative config is more/less the stuff underneath $HOME or the XDG config dir.
Turns out that a few things with fish shell don't work with a purely declarative config. (e.g. the universal environment variables are stateful).
For myself, I don't rely on writing all my configuration in Nix files. I configure fish shell using config.fish. (Though I do use HM to symlink the config file).
Ie with Home Manager, i just have Nix copy (link) my config files to the place apps look for them. I only would even consider recreating them in Nix if i needed programmatic construction of them or if i was trying to distribute them as a Nix package or something.
But for just me, and my own use - i see zero reason to recreate config files which are already managed by Nix. It's just extra work with no benefit as far as i can tell. Especially because i detest the Nix language. It also makes it very difficult to search for, because if you want to figure out how to configure X feature of Y app, you need to search for that and then mentally map it to Z Nix configuration which may or may not support it. It's just layers of needless abstraction for a single-use configuration.
home.file.".gitconfig".source = config.lib.file.mkOutOfStoreSymlink ./dotfiles/git/.gitconfig;
xdg.configFile."nvim" = {
recursive = true;
source = config.lib.file.mkOutOfStoreSymlink ./dotfiles/nvim;
};Haven't started down into the Nix abyss myself though…
Also, you can use home-manager from within your configuration.nix, that’s how I use it! Then you don’t have to deal with it separately and everything is at one place.
The thing is that while NixOS configuration.nix also contains modules for software, it primarily concentrates on global services and might not have modules for programs that are meant to be run under user.
What separates the professional from the 'personal and hobby stuff': the need for (timely) reliability, and that you want other people to be able to maintain it.
With Nix, because it's so strict, it's more difficult to quickly kludge a solution. If something doesn't work as you expect, it may demand understanding of different parts of the solution stack.
That Nix isn't widely adopted means it's harder to find e.g. StackOverflow answers for problems.
In the blogpost, an example of a problem they had to work around was that fish's configuration expected to work statefully in a way which didn't work well with Nix's Home Manager's declarative nature.
As for devops task, if it is something that you plan to do and many other users already did than it should be fine. If you are not happy with existing solutions and want things done your way, or it is an uncharted territory then you will need to go through the steep learning curve.
That said, I think the experience is demonstrative.
When Nix works, it's amazing, & you'll want to have Nix everywhere.
When something goes wrong with what you're trying to do, Nix can be very challenging, and will typically demand you have a good understanding of what's going on (with what you're trying to do, with Linux, with Nix in order to fix it.