Nix Survival Mode: macOS upgrades won't break Nix anymore
determinate.systems
determinate.systems
More background in https://github.com/direnv/direnv/wiki/Nix.
Try https://github.com/direnv/direnv-vscode or https://github.com/fehnomenal/intellij-direnv
Personally I prefer to even store regular Mac apps I download in ~/Applications, and anything I can install without disturbing system-global stuff, I prefer to do that way. It means I can migrate things completely by copying one directory, rather than using an "Installer" in the Windows tradition.
The default path could be changed too, of course, but putting it under your home directory would mean that you could only share cached packages with other people that share your username. It would also break NixOS, since that uses Nix to manage the entire system.
> It means I can migrate things completely by copying one directory, rather than using an "Installer" in the Windows tradition.
Nix ships with a specific tool for copying packages (including dependency trees) between computers, nix-copy-closure. But this also assumes that the store path is the same.
Do you know whether Nix has a proposal to fix that big ease-of-relocation issue via path rewriting? Afaik Spack package manager builds with padded paths so those can then be rewritten to practically any install path
EDIT: theres also work being done to move to content-addressed nix store, in which case changing these things would invalidate the content hash
Relocating is really only a matter of relocating prefixes.
I am not sure how it would affect a content-addressed nix store, but I'm also not 100% sure why you want a content-addressed nix store. Addressing by the derivation hash (which is also what Spack does) lets you find valid build substitutes... would you index that separately for a content-addressed store?
Is the goal to allow multiple versions of non-reproducible builds in the same store?
While we're speaking of things I need workarounds for: having my login shell be Nix-installed currently has a race condition on systems with FileVault. I have, for example, iTerm2 configured to resume on reboot and it will start before the Nix partition is mounted, thus complaining that it cannot exec my shell. My workaround entails placing a thin wrapper on the main partition that simply waits for /nix to be mounted and then `exec`ing the right shell binary depending on the invoking user.
I should probably open an issue about this, but I'm not sure it can be solved at all while `/nix` needs to be on a seperate volume.
[0] https://github.com/Cu3PO42/gleaming-glacier/blob/master/modu...
I love the Determinate Nix Installer! My interactions with the maintainer have been very positive, too. Can't recommend it enough. :)
On the other hand, Nix installs software into `/nix/store/longhash-pkgname`. By default any given package is not aware of anything else you have installed, unless said package explicitly referenced the other during build time, including the exact version of the package.
This makes it possible for every package to get exactly the version of a dependency that it's known to work with and allows you to have different packages working with different versions of libraries. This makes Nix extremely reproducible: once a package works as intended, it will almost certainly continue to work no matter what else you change on your system.
On the other hand, I have experienced issues with my system after `brew upgrade`s on multiple occasions. Particularly related to Python upgrades.
I still use `brew cask` to install most GUI applications, since the repository for those on macOS is larger.
In my configuration, I have this
homebrew = {
enable = true;
onActivation = {
autoUpdate = true;
upgrade = true;
};
casks = [ "inkscape" <other gui apps ...> ];
}; home.file."Brewfile" = {
onChange = ''
HOMEBREW_NO_AUTO_UPDATE=1 /usr/local/bin/brew bundle --file="$HOME/Brewfile" --cleanup --no-upgrade
'';
text = ''
cask '1password'
...
'';
}Here’s how I do it: https://github.com/dustinlyons/nixos-config/blob/main/flake....
I can type `nix-shell -p <package-name>` and get a new shell with that package. Maybe it's something I just wanted to try out. When I exit the shell, it's basically gone (it's still on my disk, but it hasn't changed anything in my normal shell environment).
I can have different versions of Python or Ruby for different shells. In fact, I can solve some of the "works on my machine" issues with Nix. I can put a shell.nix file into my git repository. Then when someone else wants to run my Python code, they can use `nix-shell` to bring up a shell with everything I expect to be there. I don't have to tell them "ok, first install version X of Python. Oh, you already have version Y installed? ...well, figure that out and then we're moving on to installing..." It creates a nice little semi-isolated shell with all the things needed to run my thing - and I can replicate that environment on other people's machines.
It's also a ton faster than Homebrew when installing things. I was shocked how much faster.
My advice (if you're interested in trying it): start small. You already have Homebrew. Install Nix on your machine and then forget about it - until you want to install a new package. Then type `nix-shell -p <package-name>` and you have a shell with that package. Later, you can create shell.nix files. You can commit these to a git repo so that you can bring them to a new machine.
I don't personally use it that differently from Homebrew, but there are certainly times when it's really nice to be able to install something without it causing chaos in my system and it's nice to have files written down with the things that I want installed so if I want to reformat my machine, I can get things back up nicely. I'm sure there are more use cases for its semi-isolation, but in the meantime I've been quite happy with it.
Nix solves version and configuration dependency hell problem by allowing multiple versions and configurations of the same package side-by-side. Unfortunately, it also still wants to make system-wide changes on macOS. It also has the "problem" of less wider accessibility because the functional nature is more complex.
PS: I have worked in client platform management and engineering at scale.
For context, these changes are:
1. Creating and mounting a volume at /nix for the store
2. A daemon used for isolating build environments
3. A zshrc / fish / bash hook to load a few environment variables (PATH, mostly)
4. (new!) an on-boot Launch Daemon to keep those profile hooks working between upgrades.
1. Nix installs all necessary libraries as well as part of the install path. That is - if you need two versions of a library, you aren't needing to contend with whatever is globally installed. I ran into this a lot with Python stuff, personally.
2. You can install different versions of the same software without needing dedicated version managers. This is immensely useful if you work on several different projects that might be using different versions of tooling.
--
But that's just Nix the package manager. Brew does not solve problems of OS configuration, dotfiles management, reproducibility, etc., all of things Nix solves off-the-bat.
For example, I mainly use macOS at work but NixOS for anything personal, as well as servers. I can maintain roughly consistent, through code, environments with only minor tweaks per machine. This means I have essentially the same development environment at work and at home for anything in the terminal.
Another big benefit is, if anything breaks, you can always roll back with Nix. Or, since it's code, it's very easy to start with a clean slate. I recently changed Macbooks and, with only a minor tweak to account for the ARM chip instead of Intel, I was up-and-running with my entire environment exactly as it was, with the only time accounted for being package download and installation, essentially.
--
The main problem that Brew solves that Nix does not (well, it does, but it defers to Brew to do so), is installation of graphical applications. There is an open Apple bug on this issue IIRC - I don't recall all the specific details at this moment, however.
The issue is that Spotlight doesn't index across volumes other than /, so when you symlink apps into /Applications or whatever from /nix/store, Spotlight won't recognize them.
Installing those applications via Homebrew is one way to manage it, but you can alternatively instruct Nix-Darwin to install those applications by copying them out of the Nix store rather than symlinking, which also works.
I'm not sure if Homebrew has this - but it takes me a few seconds to spin up a new Mac entirely with everything i want (minus build times lol). With home brew i was fiddling with "oh yea, this package/lib/plugin is missing" for weeks.
edit: Also, all repos i have are also consistent. Ie i build my repos with specific versions of npm, rust, etc - all per repo, all consistent across all machines, all managed by Nix. Think node-env or py-env (or whatever they're called), but Nix and cross-language instead of specific to node or python ecosystems.
Is nix like a npm lockfile then, carefully controlling specific versions of packages and dependencies on a per... what, per-machine?... basis?
Or is it kinda like a Docker container that can contain isolated programs?
Or something like pyenv or nvm that can let you switch between version sets?
The next step is to actually take that digraph and instantiate a bunch of packages on your system. Each package has its dependencies symlinked to it using env magic, and bubbles up into your shell session or whatever.
All the different offerings in the nix ecosystem are based off this idea. Home-manager focuses on user session environments, NixOS extends the concept to a linux system (e.g. systemd units are managed in the same way - its just another hashed object in the nix store that gets symlinked to systemd), NixOps lets you do it to other machines, and nix-shell lets you create per-project development environments.
Nix flakes are the next evolution in the ecosystem, where some of the "inputs" get taken out of your env where they were black-box magic and put into a flake.nix (with a pinned version locked in flake.lock) so the input package set is controlled for across builds.
> Is nix like a npm lockfile then, carefully controlling specific versions of packages and dependencies on a per... what, per-machine?... basis?
Installation and management of software with Nix is generally handled via tools I'll call 'profile managers', e.g., nix-env, `nix profile`, home-manager, nixos-rebuild, darwin-rebuild (Nix-Darwin), Disnix, NixOps, etc.
These tools all support version pinning (typically via 'flakes', but alternatives can and do integrate this functionality with various Nix profile managers as well).
These profile managers collectively operate at the cluster, machine, and user levels, but on a given machine you can also manage arbitrary named profiles, and you can additionally use Nix without persisting any environment into which things are 'installed'. In the latter case, one can still use source control to pin those environments, allowing for general-purpose, polyglot, per-project package management.
> Or is it kinda like a Docker container that can contain isolated programs?
Yes, but the kind and level of isolation that Nix provides is thinner than Docker. It's most comparable to something like Python's virtualenv or Ruby's Bundler, where packages are installed natively on the local filesystem, but managed via environment variables and symlinks.
The only real difference there is that Nix packages are configured at build time so that they're very hard to find by the dynamic linker and so they're very bad at dynamically looking for each other, providing some additional 'isolation' in a weak sense.
> Or something like pyenv or nvm that can let you switch between version sets?
I wouldn't say so, but you could manage Nix profiles that way if you wanted to. The CLI would be clunky for this at best.
As for the granularity: that really depends on how you want to go about it. If you manage a NixOS system (a Linux distribution built on Nix), you can control the versions on your system. But any project could have its own environment entirely. In fact, in any project you can have multiple versions of nixpkgs if you really wanted to.
Nix is a programming language plus utilities. It is designed for packaging software in a reproducible way (https://github.com/nixos/nix/).
Each package is called a "derivation", which is a function that takes inputs and makes output. The inputs are everything that is needed to make the output. It is "pure functional" package management - for the same input arguments, the same output will be produced. Nix is really fast because each derivation is hashed and cached and the language is lazy-evaluated.
Builds are "hermetic", meaning only the inputs specified in the derivation are available at build time. Contrast this to some packaging systems, where the build is done against some staging area where packages get installed as they are built and the output can depend on the non-deterministic order that packages are built.
Nixpkgs (https://github.com/nixos/nixpkgs/) is a large collection of recipes for existing software. It contains both rules to build software as well as "modules" to configure it or extend it. NixOS the linux distribution is also part of nixpkgs. There are lots of design patterns here and it can go pretty deep. There are also tons of hacks and patches and workarounds to make software conform to the way nix works. Nixpkgs also has a lot of useful library modules built in.
Nix is the latin word for snow. Nix "flakes" are a way to combine multiple flakes as inputs as well as pin their version. Kind of like pipenv/requirements.txt or "cargo lock" or "yarn lock" but for anything.
The output of derivations go in the "nix store" which is a path like /nix/store/<hash>/, so all sorts of software can co-exist (think multiple incompatible versions of the same library) and can be referenced in a fixed way.
Any kind of software can be packaged as a ".nix" file. It's really common to make a "shell" for software. For example, today I wanted to run GIMP (the image editor). I didn't have to install it in a traditional sense. Instead, I ran 'nix shell nixpkgs#gimp -c gimp'. That makes a shell environment for the "nixpkgs#gimp" derivation, including adding /nix/store/12naig12mnhrzpn88bvvw2vakyd18sjq-gimp-2.10.34/bin to PATH, and runs a shell script command (gimp). Nix makes sure to fetch all the outputs and runtime dependencies of the gimp package for me (which are cached locally and on the internet).
One area where nix shines is that you can write nix code to build a reproducible dev environment for your project.
You can think of it kind of like if you could use a Brewfile to build a virtualenv that doesn't touch your system at all.
I like this a lot because it makes dusting off old projects way easier. I don't have to worry about the state of my system, or reverting to an old version of ruby, etc. to get the old project running again.
Nix is also much more powerful than a Brewfile. You can override the versions of dependencies or even package your own custom dependencies and use those in exactly the same way. The learning curve is steep, but once you learn you have an incredible amount of flexibility and power all built on a robust foundation.
But wouldn't you want to update your project to a new(er) version of Ruby in that hypothetical?
There have probably been security updates along the way.
Perhaps because it's in maintenance mode and what you really need is just to be able to run it to compare its behavior with that of its replacement, or because you want to perform the upgrade by replacing code that relies on deprecated APIs in an incremental way, in a copy of the codebase that still runs and passes tests at each step along the way.
Or maybe you're a scientist and you want to run the original code because you want your tests to have the exact same performance and other characteristics as the code used in the original paper, so you want to avoid compiler/interpreter optimizations that have come out in the meantime.
Clearly you shouldn't dust off a Ruby 1.8.7 app and deploy it to prod, but what would be the harm in using a script you wrote in 2012, say, to batch process a CSV to JSON?
In fact, the project you're dusting off might just be an old revision of your current app that you already ported (over the years) to current standards, but you thought you wouldn't need that script so you deleted it 7 years ago rather than keep it up to date. if you could check out the old tag and run the script intact, that would be ideal.
The main advantage of nix against brew purely as a mac package manager is the ability to easily use versions of things that aren't HEAD (or to use multiple different versions of things for different reasons). Homebrew has a really clunky way to do that, but its much more straightforward in nix.
The factors I care about in a package manager, in no particular order:
1) How good’s the UI?
2) How often does it break or otherwise make some kind of bad thing happen when I use it?
3) How bad is my day going to be if it breaks or does a bad thing?
4) How good is the package selection?
Brew is easily my favorite, based on that.
> 3) How bad is my day going to be if it breaks or does a bad thing?
Gosh - I've used FreeBSD's package manager, Gentoo, MacPorts, apt-get, ad-hoc bash scripts, brew, nix, and even occasionally manual compilation for package management over the years, and brew is by far the one that's caused me the most pain.
Just in the last year of $DAYJOB I've probably had brew break my dev environment at least three times because I installed a new tool and it auto-updated the others when I did that.
Maybe I'm holding it wrong, but the others haven't done that to me.
Obviously that's not your experience, and I don't intend to dismiss or demean that.
It's just shocking to me how different our experiences of brew are.
FWIW nix is definitely my favorite packager manager so far. I've been using NixOS on my laptop for about a year, and nix on macOS for package management before that, and it has been shockingly robust. I do not recall a single time a tool I got installed and working via nix has broken on me.
Running a Linux desktop has been kind of disappointing in a lot of ways - while they've improved a lot in the last decade, there are tons of ways macOS and even Windows have superior overall UX.
But for robust package management, nix has been everything I ever dreamed of and then some.
1) One to manage the OS software,
2) One to manage the packages the user or users run directly, and
3) One that can be sandboxed or otherwise isolated from the rest of the system, to some degree at least, for software project dependencies.
Brew does not attempt to do #1 (that’s totally separate—which I now regard as the right way to do things, keeping the kernel et c. installer program and the Firefox et c. installer program separate is better) and is bad at #3, but so are most Linux package managers.
Most of the time when I see people having trouble with Brew, they’re using it for #3 and it’s bad for that, so it’s not working well for them. It’s a type-2 package manager (but, again, most Linux package managers are also bad at that type-3 thing).
I use asdf, direnv, and docker (for daemons—99% of my use of docker is as a sandboxing package manager) for my software project dependency management (type 3). Needing three things sucks, but each does a different thing, and it beats screwing around with per-software-ecosystem tools for things like virtual environments. I would also use them if I were back on Linux (most major distos, anyway—NixOS might be an exception)
Nix, notably, is probably the only semi-popular package management system that could plausibly do a really good job at all of #1-3. I’d probably use it on Mac and Linux both if the package selection were as good as Brew (on Mac), and I didn’t hate their DSL.
I think most popular(ish) Linux package managers other than Nix are worse type-2 package managers than Brew, and bad type-3 ones (as is Brew). I don’t just like Homebrew because I’ve not used plenty of other package management systems, though.
Things like nix-darwin and home-manager allow you to decoratively define your system config and dotfiles. Because it is a language you can override things in same place you're defining what packages or etc you want.
And if you start using `direnv` with it's flakes plugin (enabled by default when installed using home-manager!) you can define a flake.nix which will keep a lock file independent of your system's/configuration's that means anyone with Nix installed should be able to activate a native dev environment with the exact versions of node, python, rust, pnpm, etc. ready to go.
It's quite powerful once you become familiar with how to use it.
As an example, Environment Modules solves 98% of these problems and has been available for more than 30 years:
My config is loosely based off https://github.com/dustinlyons/nixos-config if this matters. Using nix-darwin.
1. https://github.com/determinateSystems/nix-installer#uninstal...
edit: tried again and got 0.14. Weird.
We're just ramping 0.14 up to 100%. So please try again in about 10 mins
1. https://determinate.systems/posts/determinate-nix-installer
2. https://discourse.nixos.org/t/nix-survival-mode-macos-upgrad...
What's complicated (and often not or only lightly documented) is nixpkgs, the package library.
But yeah, the Nix language is definitely not the complicated part. You can get the language down in a day or two with their reference doc. What is difficult is understanding the patterns to NixOS and nixpkg's. Overlays, derivations, packages, flakes.. that's where things get the most confusing.
It's a huge step away from other packaging mechanisms like PKGBUILDs but much more powerful and type-safe.
But some things just aren't thought of, like home-managers `home.sessionPath` for updating the PATH is _append-only_.. so it literally cannot be used to prepend a preferred bin path. So you follow the typical convention as you might already and include it in the shells rc with `export PATH=/custom/path/bin:$PATH`.
Which is the exact kind of safety you'd have today, so it's not worse, it's just another one of the shortcoming of the modules, as opposed to the Nic language itself. The same shortcoming would exist if you used HCL to declaratively define additional PATH paths. It's typically the module design, not the language.
But it's a real annoyance, and the kind of thing that could turn off or confuse new and casual users. And workarounds like this are not unusual on macOS, where upgrades verge on developer-hostile and user-expected behaviors are often impossible without hacks.
EDIT: User and process namespacing would help a lot as well, to improve build time isolation.
On a tangent: I’m a big fan of Homebrew, but Nix’s deterministic builds are more isolated than Homebrew’s. Last I heard, Homebrew also tries to avoid keeping around any but the latest version of a formula, which would force you to upgrade. With Nix, your installed libraries are isolated, so I can (for example) open a shell with GCC 5 and Python 2 and another with GCC 12 and Python 3.12 with absolute confidence there’s no “contamination” so to speak. With Nix you come a lot closer (even on macOS) to achieving the dream of having your entire system declaratively managed, so you can get a brand new machine, pull in your config from GitHub, and have an identical setup with minimal hassle. I think even with `brew bundle` it’s not that easy to achieve this.