Hacker News Reader

Comment by ridiculous_fish | Hacker News Reader

I installed nix on my Mac but quickly backed out due to the complexity. I assumed the nix store would just be an ordinary directory with a tool for managing it, similar to brew. I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.

What is the reason for all this machinery? I went with the recommended multi-user install, should I have just used the single user mode instead?

View on HN

> What is the reason for all this machinery?

> new Unix group

This is used for the daemon, so it doesn't run as root and expose your system to Nix build code.

> adds a separate APFS volume

I think this is required because of macOS security restrictions preventing direct modification of the root directory. The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix, and that can't really change per system because it would break caching and reproducibility. The separate volume is added to /etc/synthetic.conf so it can live in /nix.

> installs a daemon

For multi-user installs, this allows non-privileged users to add build outputs to the Nix store, which ultimately allows these users to share build outputs. More info here: https://nixos.org/manual/nix/stable/installation/multi-user....

The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix,

This is true, but in some sense Nix on Apple Silicon was a missed opportunity. It started as a blank slate (fresh binary cache) and it would've been a good opportunity to move the store to a writable path like /opt/nix. This would have solved the whole dance needed with volumes and synthetic.conf. I know that there are infrastructure issues (like Hydra using a single Nix store), but it would've made the macOS Nix story so much better.

It's a shame that only Apple can make firmlinks, because that would've been another possible solution (/nix could be a firmlink to the actual store location).

The same problem occurs on e.g. Fedora SilverBlue, because you can also not make arbitrary root directories. But at least on Linux, you don't want to throw away more than one decade of a x86_64 binary cache.

> It started as a blank slate (fresh binary cache) and it would've been a good opportunity to move the store to a writable path like /opt/nix.

That breaks caching cross-compiled outputs and x86_64 outputs, which do still run on Apple Silicon.

Which most users don’t really care about. I think it’s optimizing for the wrong use cases.

This is the main reason why I don’t use Nix on the Mac anymore, I just don’t want the pile of hacks that is necessary on my system. And I am a former nixpkgs contributor. Many people will just shrug and install Homebrew.

Actually the cache isn't that important, most binaries get rebuilt quite often due to their dependencies changing.

Although it is my hope that rfc 17 eventually makes it through: https://github.com/wmertens/rfcs/blob/master/rfcs/0017-inten...

Actually the cache isn't that important, most binaries get rebuilt quite often due to their dependencies changing.

Sure. But it's still nice that e.g. on Linux x86_64 you can use a nixpkgs commit from 2015 and get all stuff from the binary cache.

Single-user Nix installs are lighter weight, but also have a major compromise: the store is writable by the user. This means it is pretty easy for software you're using to accidentally change the store underneath Nix, and create unpredictable and unreproducible behavior.

I think the multi-user install is a better Nix experience, even if the install process is spookier.

Like is mentioned elsewhere, Docker does similar contortions to install itself. I wonder if it would have been more palatable if the Nix installer was less forward about what it is doing?

> I wonder if it would have been more palatable if the Nix installer was less forward about what it is doing?

I bet most wouldn't consider it spooky if the installer didn't print it out.

If docker started printing it, I bet lots of people would complain it seems similarly complex.

I recall the sqlite changed their tmpfiles on Windows from "sqlite" to "etilqs".

> What is the reason for all this machinery?

Nix wants to use /nix, but Apple wants macOS to be locked down and secure.

The compromise was putting /nix on a separate volume.

> I discovered it creates a new Unix group, ... installs a daemon

This is at about the same level as Docker, fwiw.

> The compromise was putting /nix on a separate directory.

Why does a separate directory need a whole other volume? These are just files, right? Why do they need to be on a separate volume?

Many binaries when compiled gets hardcoded paths compiled into them which are determined during configuration time. This means you cant use binary from a cache if the file hierarchy isnt identical. On linux this isnt a problem due to namespaces (feature originating from plan9).

OSX requires mounting root folders like /nix with synthetic.conf IIRC.

Right but the comment said they compromised on using a different directory in the end, so it could be under /opt like Homebrew, and not a root directory.

One of Nix's benefits is it aims to isolate where programs are installed; so you can have multiple versions of the same program available without conflict. (e.g. programs built with different feature flags, or multiple minor versions).

Nix achieves this by storing packages in a path as some hash of its inputs. -- This then allows either compiling or downloading a package, with confidence that it will behave the same way regardless.

But, since the files were under /nix, if you put them under /opt, then you wouldn't be able to make use of the compilation caches for /nix.

But, since the files were under /nix, if you put them under /opt, then you wouldn't be able to make use of the compilation caches for /nix.

Until a very short time ago there was no binary cache for Apple Silicon Macs.

There are for anything you run as x86_64 (using Rosetta). It's also important if you're cross-compiling anything.

Sure. But I think 90% of the Nix on Apple Silicon users does not compare about cross-compiling or running x86_64 binaries with Nix. So, everyone is worse off for supporting minor use cases.

Yes but they found a solution which works for 100% of users.

I don’t really understand the complain in this thread. Using an APFS volume just works and is entirely transparent to the user.

People could just learn instead of complaining they don’t understand.

I guess I don't understand what you meant by 'The compromise was putting /nix on a separate directory' then?

Oops. Wrong word. I meant "separate volume".

/nix could still be used if it's on a separate volume. Which still seems less nice than just having /nix on the same volume.

> it could be under /opt like Homebrew

You can install Homebrew other places, too, for that matter.

I'd only tried NixOS (bounced off, couldn't get X-Window working even following tutorials to the letter) not Nix on macOS. "Must be installed in a specific, root directory" is a hard no for me, when it comes to add-on package managers. That's one hell of an odor.

Homebrew sometimes pretends you can install it somewhere else. Like most things with homebrew, if you try, you will quickly realise this is not regularly tested.

> That's one hell of an odor.

That’s very much fine. They do that to be able to share the cache between system. There is nothing magical about the root folder by the way. It’s just Apple being annoying.

It's because Nix wants to install into /nix. Once upon a time doing "sudo mkdir /nix" wasn't a problem, but recent macOS releases have made that very hard.

Nix could switch to an alternate location on macOS (e.g. /opt/nix) but that has a lot of downsides for interoperability with other systems.

Yikes, I had no idea. What a tough pill to swallow all because the directory nix runs out of is hardcoded.

On Linux, users can put their Nix store in their home directory or other places and at run-time Nix remaps the directory using user namespaces. Unfortunately this isn't workable on macOS: the kernel doesn't support the features we need.

Using /nix and a separate group and daemon means the store can be read-only and be protected from modification in several ways. This is pretty helpful, as a lot of tools try very hard to write "next to" where they are installed -- corrupting the Nix store.

I sort of wonder if it would be more palatable if the Nix installer was a bit less in your face about what's going on? This would be similar to how Docker's works.

It would be less palatable when I found out. The group is fine. Why the daemon when other package managers use sudo is unclear. Even Homebrew moved to /opt.

Other package managers are okay with requiring sudo because they install stuff globally. Nix doesn't have that restriction, you can use it for local stuff, temporary shell environments, etc. So you need non-admin users to be able to use it too, and even admin users need to be able to use it without using sudo. For example, when entering a nix shell, you don't want the shell to run as root. Or when using direnv. Or just when using it as part of your build system.

Nix is a package manager, yes, but it's more than that, it's a generalized build system.

Nix isn't the only package manager for home directories or source packages. Working in a home directory doesn't require sudo or a daemon. sudo doesn't mean run everything as root.

You missed the crucial point:

> So you need non-admin users to be able to use it too

The build daemon and the user are used for privilege separation. The separation goes both ways. Users can't write directly to /nix/store and Nix can't write outside of /nix/store during build.

If anything, it's there to make things less invasive. It's nothing like the Docker daemon, which is a proxy for root.

Additionally, the daemon doesn't do anything unless users request that a package be built.

Other package managers have privilege separation without daemons. I don't need non admin users to run it. And I could configure sudo to let them if I did.

This is inherent to having binary packages. Binaries compiled to look for libraries in one location cannot just be copied to a system with a different layout. Nix also can't just use the OS conventions, since part of the point of Nix is that it does not use the global system state and only its own isolated, reproducible world. So then they had to make a decision and decided to just use /nix everywhere which worked fine for some years. And now MacOS has changed, choosing something else would invalidate all historical binary packages.

It's not hardcoded, but you end can't take advantage of the binary caches if you change the directory. A company could certainly create their own binary cache and distribute that to it's users.

Not entirely true, there are many ways in which you can use a custom location and still take advantage of the binary cache. You can do it with chroot, file system namespaces, bind mounts and so on. There's also a nice user friendly tool that does exactly this [1].

[1]: https://github.com/DavHau/nix-portable

> What a tough pill to swallow all because the directory nix runs out of is hardcoded.

Yikes? Well then... how would you solve remote binary caching with something like Nix on a platform such as OSX without userns remapping support?

It’s not actually hardcoded, but you would have to compile everything from scratch as the hash would depend on that as well, throwing out the whole binary cache.

Actually, there's quite a few hard-coded references to /nix in nixpkgs I believe. It's possible but a little more work than just changing the prefix.

There may be, but at least one BigCorp deploys Nix to a different prefix.

The install process was simpler before recent OSX updates.

> I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.

Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?

> What is the reason for all this machinery

Enforcing reproducibility basically.

I don't know all but...

The daemon and APFS volume otherwise readonly /nix that only the daemon can write to can't be created.

The group is probably for the daemon to be able to write to /nix.

The path /nix is important because the remote binary cache paths will miss otherwise and you'll compile everything from source.

> I went with the recommended multi-user install, should I have just used the single user mode instead?

I'm guessing it would work in the way you want, but I always opt for the daemon.

> Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?

Fair observation. I installed nix as a prerequisite for DevBox, discussed here https://news.ycombinator.com/item?id=32600821 . I thought DevBox sounded really cool (and still do!), but the Quickstart took frustratingly long, and ended up not working. Faced with the prospect of debugging it, I opted to cut my losses and uninstall it instead.

That said I'm very much open to trying nix again in the future. Also I want to acknowledge how much effort went into getting /nix to work on the Mac; it appears that was a heavy lift indeed.

The meta answer to the question is this info belongs at the non-existent doc - https://nixos.org/manual/nix/stable/installation/macos.html, but no-one has put it there yet.

It's because builds are completely reproducible, and therefore they have to be built in the same place (in `/nix`), and you can't create directories off of `/` on modern Mac setups.

APFS volume is required to have a read-only nix-store at /nix. The daemon is only required in multi-user installation, and you can opt out of it (by using a single-user install). New groups are for letting people submit jobs to builders in multi-user mode.

Note that the multi-user installation is these days the only suggested approach by the Nix team for new MacOS installations - the old instructions for single user installation on MacOS have been removed from their site.

Reply on news.ycombinator.com