The Determinate Nix Installer
determinate.systems
determinate.systems
I wonder if you took a look at some of the modifications done by portable-nix (https://github.com/DavHau/nix-portable), most important ones being:
a) Allowing user to choose the location of the nix folder (for example $HOME/.nix) by using bwrap or proot
b) Same binary for Win/Linux/OSX on x86_64 (Also see https://ahgamut.github.io/2022/07/27/ape-rust-example/)
And then, somewhat unrelated: c) Can it be used within NixOS as nix.package = ?No, it can't be used that way on NixOS, but we could put something out that sets up a similar experience on NixOS! Good idea!
Right now i have this adhoc bootstrap process and it feels a bit silly. I get it's slightly different to getting Nix itself installed, BUT, it's also somewhat related from a user point of view. It would be interesting to see something handle this out of the box.
However, the documentation team still exists and meets— there are still people dedicating much effort to improving the official and community documentation in a concerted way.
I feel that those events are more an interpersonal wound internal to the community than they are the death knell for the overall effort to renew the Nix ecosystem's documentation, or a sign that Determinate Systems is not at all invested in the health and integrity of the community and its volunteer organizations. I think turning wholly pessimistic about those things at this point would be a big overcorrection.
Edit: Oh, just saw the link to GitHub with the supported systems, missed that the first time I read the page.
(1) https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...
Regarding your argument against Bash, isn't it true that this is just a different version of the same bootstrapping problem, and that downloading a temporary but specific Bash version would get around that? (I agree that working with Bash leaves a lot to be desired. But it's also more accessible to end-users to hack on, observe, and learn from, than a compiled Rust binary is. And tools like Copilot and ChatGPT and https://www.shellcheck.net/ make working in Bash MUCH more painless, I've found.)
Couple more questions: 1) Is your installer idempotent? (I think it is, since you track what's been done in a JSON structure; just confirming! If the JSON file is lost, are you still idempotent though? lol) 2) Are you a profitable company or just a group of Nix believers?
On NixOS: I generally agree, though it can be tricky to get that done with NixOS. I bet we can make some inroads there.
Easy questions:
> idempotent
It isn't currently idempotent, but we're working on "curing" which brings some idempotency to it, and allows for "rescuing" installations that aren't working correctly. Curing will also support creating a JSON document for installs that don't already have one.
> profitable company or just a group of Nix believers
We are a for-profit company, and we are a group of Nix believers :).
Bash:
Bash is really hard to get right, and really hard to make fast, and really hard to extend its featureset to more complicated use cases. We could "just" download a bash, but when we're trying to navigate disk encryption and APFS volumes and mounts ... it is quite a lot easier to write and test that code in a "bigger" language.
Also note that it isn't just Bash that we'd need to download, but coreutils and other utilities that Bash is really built on top of. We tried to fix a pretty simple bug last year in the Bash installer, and it took two weeks to validate a relatively straightforward, one line change was valid on all the platforms we target.
This clarifies a lot of what confused me in TFA. I maintained millions of lines of bash code, some of which is decades old and run into only a single bash incompatibility that wasn't fixable with shopt. On the other hand, incompatibilities in programs bash is running are rampant, particularly between GNU and non-GNU systems (and even within GNU there are more surprises than you might think). The phrase "different bash implementations" was a major red herring for figuring out what you were talking about.
We're always looking for good writers, Rust devs, and security folks. People who are in to these things should reach out with their resume: careers@determinate.systems. Note that existing Nix expertise is not required.
This really is the problem with bash. It doesn't have a stdlib. Posix tried but didn't succeed in sufficiently standardizing userlands. Bash is friendly and easy to hack on, but hard to manage at scale, for this reason. And this crops up all the damn time when you're working with compiled languages. Nobody wants to have two languages. But if one of your languages is compiled, then nobody wants to write and maintain literally everything in that language. At some point you need scripting. Bash is usually what's reached for. But OSX doesn't ship a recent bash. And your coders are probably running OSX. So you have to write scripts that target ancient bash so the devs can work locally, but also work in the cattle farm. You'll spend lots and lots of time fighting it. Or have ridiculous 'setup' docs that will trip up newbies and you'll have to troubleshoot their setups when they inevitably not follow them right or they just don't work leaving their personal systems in an unknown state.
Or just, yannow, use a scripting language to script with instead of a shell. Or, hear me out here, use the 'scripting' language to build your whole app. You very very very probably can. Ruby's object oriented and is way friendlier than bash, ditto for Python.
I haven't been working on the detsys rust installer, but I think I'm the person who's contributed the most to the official bash multi-user installer since Graham did a lot of work on it several years ago.
I ~like Shell, and I share your concern about it being less hackable. But the project has also been slowly bleeding goodwill around the lack of clean uninstall/reinstall for years, and I will make my peace with the compromises it takes to get there.
The problem isn't so much conjuring good shell expressions. Nix leans a lot on Shell, and I feel like the community has a fair number of people around who know Shell fairly well. The problems are more like:
- limited/tedious support for robust error-handling and recovery
- needing a significant amount of new code to handle removing what Nix adds (even though I've done a lot of work to develop the patterns to support eventually getting here)
- a fairly fat long tail of problems caused by things like edge-case environment/state differences (due to platform, due to users having surprising environments, etc.) and even flaky behavior on some systems
- screwing up the installer is fairly high stakes (and it's very common for even one-line changes to break the installer in some way for someone somewhere and end up needing to get reverted for a rethink)
- testing the installer is tedious (it's gotten a lot better over the past ~2 years--but that testing is also just covering the golden path), so iterating is slow
- writing and reviewing installer changes tends to entail (and sometimes suffer from a lack of) really broad knowledge of different platforms, shells, utilities, options, permissions, and user configuration ~patterns
Because the stakes are high and review is hard and testing is hard and iterating is slow, IME most of the people who are enthusiastic about working on it are naive about these headwinds. This might lead them to, say, submit a PR bigger than anyone will have the stomach to review/merge--or not have the time/energy to slog through weeks/months of review/edit/test iterations it takes to build high confidence in a big change. It's also a big drag on the number of people who have the stomach to continue making substantive installer PRs after they land one.
Before detsys started looking at the rust installer, I had picked at a POC for just bundling a bash and everything we touch (iirc at least coreutils, diff, patch). If their effort failed, that's definitely the approach I was going to recommend next to reduce the frequency with which we get bitten by a utility quirk/change. But that still would've left quite a bit of work to do to support uninstall and idempotent reinstalls.
I'm personally limiting the amount of non-fix work I do on the official installer to avoid falling into its gravity well and completely owning it. Once more, if any combination of the technical changes here are sufficient to multiply the number of people willing and able to touch the installer, I'll make my peace with the tradeoffs.
Do you feel similarly about the Nix setup scripts that the installer works to ensure are sourced by user shells? I'd kinda love to see those go and be replaced by a little program that sets up a Nix environment and then launches a shell for you, kinda like the `login` program. The shell-based approach has made supporting incompatible shells kinda nasty and clunky.
I've definitely felt like the Nth-shell situation is untenable and that we'll either need to abstract a definition to generate the hooks from or find some other ~general/universal solution.
I like the way this suggestion sounds with respect to getting out of that maintainability problem, but I'm not sure if it'll work quite right?
Here's one crappy rabbit-hole we've gone down:
Since macOS changed its default shell to zsh, they've been overwriting /etc/zshrc with every update (this evicts Nix's hook, confuses/scares people, and makes them try to reinstall; note that they aren't using the Relocated Items treatment we'd expect from how they do this with most stuff in /etc/ including /etc/bashrc while it was still the default shell).
An enterprising zsh user PRed a fix that moves the hook to /etc/zshenv. macOS doesn't ship /etc/zshenv, so they don't overwrite it on update. It seems to work at a blush, but Nix and everything you install with Nix ends up at the end of your PATH instead of the beginning, so everything that depends on a PATH lookup will keep using your system utilities.
Why? IIRC because the /etc/zshprofile macOS ships will do "eval `/usr/libexec/path_helper -s`", and path_helper will add things from /etc/paths and /etc/paths.d to the left of the existing PATH. I suspect if we inject a binary or even a script to preconfigure this environment, the call to path_helper in the zsh init process is still going to leave everything we added at the end of the PATH. We could in theory add an /etc/paths.d/nix, but path_helper appends this after everything in /etc/paths, so it has the same problem. We could stick our Nix-related directories at the top of /etc/paths, but macOS does ship this file so we have no reason to have faith that macOS updates don't overwrite it or won't start doing so at some point.
(We might also be able to move this to /etc/zshenv and also define a `/usr/libexec/path_helper(){ ... }` function that behaves like WE want--but that could obviously have downstream effects on other code in user profile/init scripts that uses path_helper. No clue if this is common.)
edit: i went ahead and uninstalled, and reinstalled via this, and WOW that was fast
Is the end result of installing via the bash install script and this new installer the same? Can I use the new tool to uninstall nix installed via the bash script?
Our installer also supports some environments that the upstream installer has a harder time with.
The new installer can't _yet_ uninstall a Nix installed from the upstream installer, but our upcoming work on "curing" will bring that.
Nix-Darwin OOTB would be absolutely killer imo, since it provides such a great experience once it's in place.
What's nice about Nix-Darwin for me is mainly how NixOS-like it is. I like keeping its config under /etc and treating systems using it like NixOS. There are a few modules that I think are pretty impressive and atypical for the Nix world, like the Homebrew module. I'm not sure if HM has something comparable.
I agree that brew's cask support is generally worth using even if the rest of brew is best avoided in favor of Nix or other package managers.
I also found it humorous/ironic that for something that is all about like... deterministic and reproducible or whatever: https://github.com/NixOS/nix/issues/458#issuecomment-1019743... it touched the system in a bunch of places and was a zoo to uninstall
Some of this, though:
> touched the system in a bunch of places
is probably unavoidable. For a daemon to persist, you have to plug into launchd. To get a writable /nix, you need a special volume. To plug into all the shell environments and set appropriate environment variables, you need to configure a bunch of shells.
Some of that can be improved, for sure. Some of the logic for setting Nix profiles could be moved out of shell scripts, and then only a few static environment variables could be set in /etc/environment instead of modifying shell startup configurations to source a special script, for example. But the messiest part of a tool like Nix is always going to be where it plugs into some other system which has none of Nix's guarantees and isn't under Nix's control.
An improved installer (maybe just like the one in the OP! I haven't tried it yet) is probably the best that can be done here, in automating the handling of those messy points of integration. But it will be a matter of hammering out bugs over time like for any old installer, rather than something that relies on the kinds of guarantees Nix tries to make for its own internal operations. An uninstaller for Nix is something that can (and should!) be done well, but not by simply relying on Nix's usual guide rails.
On Linux there are far few steps: - delete /nix - delete build users (could be removed in the future) - delete she'll script files in /etc/profile - delete systemd nix-daemon service - delete symlinks in user directory
I'm mostly interested in managing my home directory to start (replace gnu stow for dotfiles), but also to manage all my other software in a way that ports nicely between systems and OSes (skip over brew, flatpak, apt, etc.). I think this means home-manager, right?
Should I just start here with Determinate Nix, or somewhere else?
1. Start learning with https://zero-to-nix.com/
2. Take a look at https://github.com/nix-community/home-manager
Good luck!
Nix is interesting for me from an immutable point of view though (mainly NixOS).
I've only used nix on NixOS, so I don't know about installing nix on top of an existing (non-NixOS) OS. A collegue is running just home-manager on ubuntu, with some success.
In addition to that, check out this Shopify video series:
- https://www.youtube.com/playlist?list=PLRGI9KQ3_HP_OFRG6R-p4...
- https://shopify.engineering/what-is-nix
I share a bunch of configuration between my M1 Macbook and x86 desktop in my office. I use home-manager with config that is organized like nixos/*, macos/*, and common/*. The first two just import the latter. Here's a link if it's helpful:
Of course you can dive straight into the whole thing if you would prefer. I've tried that too (in a former chromebook). NixOS is pretty eye opening.
It's more NixOS oriented, but most of the knowledge is obviously transferable to a bare-bones (if one could say that) Nix installation on top of other OSes.
With regards to all the Nix/NixOS documentation already out there, I'm trying to keep my stuff extremely straight-forward and rather simplistic (but not superficial), so the people outside of Nix (and even outside of Linux) may give it a solid try.
Just to note, if you wish to subscribe to the newsletter, it's broken now, gonna fix it in a few days.
Also a question to @grhmc: would you maybe have some doc that describes how to UN_install nix via your installer on macOS?
I’ve deleted Homebrew from my system and I’m now using the same home-manager setup for my macOS laptops as well as my WSL installation on windows.
It’s so great to have the exact same tools and shell configuration installed everywhere.
I have not used flakes yet but if that will help me deal with the differences between my systems I’m definitely for it!
E.g I have a work Mac, personal Mac, personal PC with WSL.
When on my work machine, usernames are different and my Git config should identify me as me, and I shouldn’t have access to my personal credential for personal GitHub, RSA private keys for personal servers, etc.
But all else should be the same.
Mostly there but the credentials part I am still figuring out.
If you want to check it out, it’s here: https://github.com/leonbreedt/dotfiles
Won’t work for you out of the box tho as it depends on private repo for sensitive configs.
1) install nix
2) run setup
Looks like we might be able to get that down to just step two thanks to y'all.
I know the primary dev on this project does use wsl, so hopefully this'll make sense to them.
Docker (and containers in general) will provide filesystem, process and network isolation. Your process can pretend it has the whole machine for itself (and even a different linux distribution) even though it's not true.
A Nix package will not do any of that. The package manager solves the problem of managing application packages, versioning, and dependencies; and doing so in a way that's 'immutable'. There's some filesystem abstraction (specially if you use nix-shell) so you can pretend that a particular version is the only one that's installed in the machine, or that a package is installed when it's not.
There's complexity but there's a reason for that.
They do share one use case: run software without having to install it.
I think Docker mostly is about running services in an isolated manner, and is easy to distribute (because its images will run the same way in each host). -- But, I do see some CLI tools provide a docker container, too.
Nix, with its flakes, supports a `nix run` command which is very similar to `docker run` (if you're thinking of "run without installing" rather than "process / network isolation"). -- In that sense, it is like 'Docker, but without containers'.
e.g. you can run `nix run nixpkgs#helix` and try out helix, or even `github:helix-editor/helix` to build & run helix from its source.