Nix 1.0 released: purely functional linux package manager | Hacker News Reader

Nix 1.0 released: purely functional linux package manager

hydra.nixos.org

120 pointsmoonboots59 comments

The functional approach seems to work great for sandboxing binary artifacts of packages, but configuration files and user data are more difficult to handle. Daniel Burrows, a debian apt developer, wrote an analysis of Nix a while back[1], and he brought up the problematic example of upgrading database configuration file formats. It seems difficult to seamlessly upgrade and downgrade these types of files whose semantics and syntax may change between versions. When these conflicts occur in Debian/Ubuntu, the upgrade process blocks and the user is prompted to resolve the 3-way merge.

I think packakge managers should only handle the storage of static files like executables and shared libraries. Forcing the user or another process to handle configuration files and startup/shutdown hooks has additional benefits such as making these files easier to store in version control.

[1] http://lists.debian.org/debian-devel/2008/12/msg01027.html

That link is actually much more concise, thanks!

In NixOS (the Linux distribution based on Nix [1]) we use the functional approach for managing all the "static" parts of the system, i.e. packages, but also most configuration files in /etc - these get build by Nix functions in much the same way as packages. This is good since it means that (say) a rollback of the system will cause it to revert to a consistent combination of packages and configuration files. (So you don't get an Apache httpd.conf that refers to some feature that doesn't exist in the Apache you just rolled back to.)

However, when it comes to stateful stuff such as user data or databases, the funtional approach indeed doesn't really apply very well. NixOS manages state in an imperative manner in the same way as every other distribution. (For instance, NixOS uses Upstart jobs to create/update state.) That means that a rollback of your system won't rollback (say) your PostgreSQL database, unless somebody made the Upstart job of PostgreSQL do that. So when it comes to state, NixOS isn't better than other distributions, but it's not worse either ;-)

[1] http://nixos.org/, http://www.st.ewi.tudelft.nl/~dolstra/pubs/nixos-icfp2008-fi...

Thanks for the response. I agree that some functionality currently handled by package managers is hairy, even for other package managers. To some extent, I don't think this functionality should be managed by package managers. It would hurt usability because the user would need to set up configuration files, hook the package into the init/upstart system, and fix breakages across updates. On the other hand, I've never been a fan of running 'sudo apt-get install apache2' on a development machine and exposing a public port and running an application I don't always need every time I boot up.

The problem with the nix package manager that I see is it doesn't integrate into a hierarchy of any kind thus no partitioning of data. Also the nix language is very obtuse and domain specific(which is a good thing as well as bad). I plan on creating a meta package manager that is functional but allows output of arbitrary formats such as rpm, deb, etc. I also think integration is a separate but just as important aspect and hope to use mruby, rspec and augeas(or something else as it seems the parsing format is yet another NIH wheel).

Ok maybe i am missing something, but it seems to me you may want some mutability in a package manager. For example, updates that are simply bug fixes and security fixes but are backward compatible should be substituted for older versions.

Suppose you have a networking library, which is used by several applications. All of a sudden you find a security hole in the networking library. And then you fix it. But the fix does not affect any of the published APIs or functionality and is thus completely backwards compatible for all legal uses.

When you fix the library you release a new version of it. Shouldn't that new version be automatically used for all applications that use the library? If you have an immutable packaging system, all applications will still use the old version of the library and they would have to have their packages explicitly modified to use the new version.

Now this may work for super high value and super secure systems, where someone has the time to individually test the new version of the library with each application that uses it and then individually update each application.

But for the usual desktop system this sounds like a recipe for having a bunch of out-dated hole ridden software, where multiple badly maintained applications still carry known security holes from many years ago; or, in other words, this sounds very much like Windows.

This is a potential problem, but I think in practice backwards compatible updates and patches for Nix could work very similarly compared to a distro like Debian. For example, when openssl releases a security fix, Debian uses its large, mostly automated test suite [1] to verify that this new update does not break compile and (to a lesser extent) runtime compatibility and correctness.

Instead of an implicit promise of compatibility, a Nix continuous build and testing system[2] could explicitly mark openssl-dependent packages as compatible by bumping their versions. The end user would need to update a lot more packages, but these cost could be reduced by binary diffing old and new packages, which should be almost identical with the exception of dependency metadata.

[1] http://www.youtube.com/watch?v=3aoDjW1bjxo [2] http://nixos.org/hydra/

In practice it gives you more options for a price of keeping more junk on HDD. After a security update, all application hashes are calculated using the new default version of a library. So, once you launch an update you get a fresh package set linked against the latest version of the library. For large applications used by many people you are likely to get a small binary patch from the previous version, but this is just an optimization. So the default is just spending a bit more resources on the same behaviour.

What does change, though, is that you can opt to pass another version of the library to a specific application manually. If update from libA v1 to libA v2 breaks AppX, you can easily build AppX-with-libA-v1. It doesn't make any sense for security updates, but for major interface changes it sometimes provides a smoother migration path.

I think that this is a stretch of the term "purely functional".

The relevant property is actually immutability, which looks nice.

While immutability and functional programming often go together, they aren't the same thing.

It is also about the way packages are specified:

"Nix is a purely functional package manager. This means that it treats packages like values in purely functional programming languages such as Haskell — they are built by functions that don’t have side-effects, and they never change after they have been built"

(from the About Nix section on top of the page)

There's also referential transparency.

For programming languages, that means the result of a function is dependent on it's inputs ONLY. The function can not use any unspecified inputs like system time or an internal state.

In Nix, that means a package can only depend on dependencies which are specified beforehand. It's nearly impossible to use libraries or binaries which you did not specify.

That behavior makes it easier to reason about your whole system, like determining which packages are unused. This is similar to the advantages of referential transparency in functional languages

> that means the result of a function is dependent on it's inputs ONLY

Not quite, it means that a function call can be replaced by its value (A function can depend only on its inputs, yet write to a global variable, for example).

The about-nix page is rather helpful on why you would want to use nix:

http://hydra.nixos.org/build/2609700/download/1/manual/#id49...

From my cursory glance over nix's advantages, it sort of looks like the vagrant deployment system, but for package management. Or maybe something akin to Chef and Puppet (of which I have no familiarity with whatsoever).

So what is the difference to GNU stow [1], if we ignore the 'works a bit like Haskell' description?

I like these kind of snapshots, the ability to install multiple variants. But it's an old hat, no?

1: http://www.gnu.org/software/stow/

Nix is a package manager in the same sense as apt and yum. It resolves dependencies, it doesn't just install software.

I don't understand this. In particular I don't understand the sense in which it is "purely functional". Surely a package manager needs to mutate the state of a system, which seems to be an inherently non-functional operation--? If I have a system S1, and then I do S2 = install(S1, "firefox"), does S1 still exist?

This bit is also interesting: "Runtime dependencies are found by scanning binaries for the hash parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works extremely well."

I think of it as git-for-system. All the data lies in the store/repository. Nix offers you to pick different isolated branches as needed.

Installing the "same" package doesn't overwrite previous versions -- all packages are persistent -- so dependencies continue to work.

It is the mutation of packages, silently breaking API and ABI compatibility, that is the bane of distro management. Nix is a great attempt to address this.

That was solved decades ago with static linking. The bane of distro management is keeping everything as up to date as possible without breaking things.

And the bane of static linking is that a vulnerability in say libc will require the re-compilation & upgrade of a gazillion packages.

With ubiquitous high bandwidth and huge disk drives, that doesn't seem to be as much of a problem as it used to be.

Yes, but you are still at the mercy of whoever is statically building (and packages) your binaries. Bad for binary linux distributions, worse for proprietary software.

Nix model is that you work with expressions that allow to build packages from source. Binary packages are provided in format "if you performed build described by hash X you would get the contents of this archive".

For proprietary software, you have to consider the binary as-distributed "source" and build a version with proper dynamic linker an library path. Some third party binary wouldn't run as-is on NixOS, but there is a package that sets correct linking settings post-factum.

Exactly. And is the result different from keeping old versions of libc around and linking all programs you haven't updated against it? From the description, that's what this system does.

If dependencies never change in place, you're emulating static linking with a dynamic linker.

Yep S1 still exists. Nix does not do destructive updating. When you install or update a package, you'll always be able to revert to the previous state. The best explanation of that process can be found in the original paper:

Nix: A Safe and Policy-Free System for Software Deployment - http://www.st.ewi.tudelft.nl/~dolstra/pubs/nspfssd-lisa2004-...

I really hope nix will become standard practice, it has some significant advantages over existing package managers.

Purely functional does not mean "has no state at all". This meme needs a stake stuck through it, though beats me how to do it. Functional languages exist and do real work.

In this case it means that packages are handled as functional programming languages handle data values. New ones are created from old ones, and unreferenced values are cleaned up via garbage collection.

It doesn't seem unreasonable to think that "purely functional" means "consisting of pure functions". Why not just say "functional" if that's what is meant?

The aim of Nix is to make function from dependencies to resulting packages a pure function. Nix seems to have achieved it with respect to set of packages installed on a system; the most problematic thing now is that some packages manage to detect CPU version at build time... Fortunately, most of them allow to disable this feature.

I may be naive in asking this, but WHY? Why is being "purely functional" an advantage for package managers?

Well, the obvious answer is that it is an advantage for the same reason that being purely functional is an advantage in other situations. More specifically, the benefit of having no side effect in the package management is that it greatly simplifies the management of multiple versions of a given package. It also simplifies the rollback of an unwanted operation in most cases.

http://nixos.org/nix/ summarizes some of the advantages of this approach. To summarize, it promises to be fundamentally safer and more flexible than traditional package managers. Package management, IMO, is an area where a "purely functional" approach could really shine, since correctness and replicability are so important.

To everyone who "doesn't understand", go read the "about Nix" section of the link.

Someone on reddit showed us this Gnome project : OsTree https://live.gnome.org/OSTree/NixOSComparison

Since Nix keeps multiple versions of almost the same binaries, it seems to me that something like ZFS, where several files can share the same physical block on disk, is the perfect match. Is that something you recommend when using NixOS?

Does anyone have experience using Nix (or NixOS) for practical purposes (instead of just testing it out) that they'd like to share?

At LogicBlox (http://www.logicblox.com/), we use it to deploy our customers' applications both for testing and production. We also use hydra (http://hydra.nixos.org/) to run our buildfarm. We're soon going to use NixOS for our deployments. If it wasn't clear from that, we're fans... :)

I use it on my notebook and it is the only OS I ever booted on it. Works fine for me, but I do like to do strange things with my system. This means that I do value the ability to revert a problematic change and I am OK with configuring things via low-level configuration files.

I'd really like to see an example of the "Service deployment" approach discussed in the overview.

Can anyone explain advantages this might have over, say, Synaptic?

As far as I know, Synaptic is not able to handle multiple versions of a given package simultaneously.

I've been eying this as a replacement for Gentoo. I like it well enough for the configurability, but I tire of the way I'm rolling the dice every time I update a package. (Binary distros like Ubuntu have this same problem, just less so.) The primary use case for me would be enabling me to stay cutting edge on software, while making distro upgrades reversible. If my new kernel blew up my system, I don't go scrounging around for boot disks, I pick the previous image on next boot.

I don't know of any other distro that can pull that use case off right now. Or any other OS, really.

I don't see how this could be better than portage

    If my new kernel blew up my system, I don't go scrounging around for boot disks, I pick the previous image on next boot.

Debian and Fedora's package managers both keep the last three kernels operating and in grub.

(This is a hassle for me since Fedora has had a long string of updates since 2.6.36 which was the last one that gave me working r/w HFS+ support, and i have to keep removing the second-to-last kernel manually with the package manager.)

FYI, there are a few yum options you can use to tweak this behaviour to how you want it to be, including 'protected_packages' and 'installonly_limit', described in yum.conf(5).

Oh neat, thank you.

In NixOS it is about all the things on the system - if updated X.org video driver behaves badly I can simply switch to an older configuration and look for a solution when I have time.

That was an example. A bad one, I see. But it applies to any foundational library, not just the kernel. What's special-cased for the kernel in Debian et al is normal operating procedure for Nix.

Atomic upgrades. Run an update, interrupt it at random time, reboot. With Nix, there is a very limited set of problems you can get due to that. FS corruption if you do a hard reboot, filesystem mounting problems if you changed your partition layout. But you cannot get a problem with a new libc being partially deployed - you either get new version of everything or old version of everything.

First yum - "a package manager written in Python" and now Nix - "a purely functional package manager". Well, I for one do not care the least which language or paradigm my package manager is using. What I do care about is compatibility, reliability, speed and correctness. How is Nix improving the situation for the user?

Immutability is often helping correctness, and for the user, a simple way to rollback (as mentioned earlier here). I'm trying to use nix as a main system so it's still hypothetical on my side. That said, nix dev team built a continuous integration system for OSes using nix, that can automate system builds on many arch, it's a good sign.

"You can have multiple versions or variants of a package installed at the same time. This is especially important when different applications have dependencies on different versions of the same package — it prevents the “DLL hell”."

"Nix helps you make sure that package dependency specifications are complete."

"Nix has multi-user support. This means that non-privileged users can securely install software. "

"Since package management operations never overwrite packages in the Nix store but just add new versions in different paths, they are atomic ... And since package aren’t overwritten, the old versions are still there after an upgrade. This means that you can roll back to the old version"

"In addition to downloading binaries automatically if they’re available, Nix can download binary deltas that patch an existing package in the Nix store into a new version."

"Nix can be used not only for rolling out packages, but also complete configurations of services. This is done by treating all the static bits of a service (such as software packages, configuration files, control scripts, static web pages, etc.) as “packages” that can be built by Nix expressions. As a result, all the features above apply to services as well: for instance, you can roll back a web server configuration if a configuration change turns out to be undesirable, you can easily have multiple instances of a service (e.g., a test and production server), and because the whole service is built in a purely functional way from a Nix expression, it is repeatable so you can easily reproduce the service on another machine."

This sounds nice, and it's a direct quote from the product page, but I can guarantee that Nix does nothing of the sort. This would need to be addressed in ld.so. It's simply not possible for a packaging system to solve this problem on a typical unix system.

0install does that by tweaking PATH, LD_LIBRARY_PATH and similar variables. That way, each package gets visibility on the dependencies it requires (in exactly the desired version).

Environment variables won't work. They're either reset for security (when sudo is invoked, for example) or they bleed inappropriately into sub-processes. There is no reliable way to solve this using environment hints to the dynamic linker. End of story.

With sudo, you just set the variables in the child process (after sudo, not before).

Variables can indeed bleed into sub-processes if you're not careful. If the sub-process's command is declared as a dependency then 0install could reset the environment automatically when starting it. Currently, it instead finds a set of libraries that work with the process and its dependencies (which is safe, but could make some cases unsolvable in theory, if a program wanted to run a subcommand which used an incompatible library).

Direct support from the dynamic linker would be nice though.

Sudo was an example, and is certainly not the only place this problem occurs.

Variables cannot be set in the child process; the linker will have already run by the time you can access your environment. Furthermore, this scheme destroys the utility of LD_* variables, which itself is a non-starter.

All setuid binaries will be incompatible with any scheme which involves environment variables controlling the linker -- that's a core element of unix security architecture.

Over here [ http://news.ycombinator.com/item?id=3995200 ] someone who I believe is the author suggested that this thing uses DT_RPATH, and I've outlined the problems with that mechanism as well.

Perhaps I'm coming across as a curmudgeon, if so I apologize. But I've worked on this problem quite deeply and there's just no way to reliably solve it. Modifications to ld.so are an absolute requirement.

Feel free to try it out to see that it does, in fact, do this. No special magic is needed (most of the time): for instance, for dynamic library dependencies you can just set the RPATH at link time to indicate precisely in which directory an application should look for its dependencies.

No, it won't work. This is my area of expertise and I can unequivocally state that it is not possible to solve this problem on a unix system. The linker infrastructure does not exist, and here's why:

RPATH (or better RUNPATH) has intractable conflicts in library dependencies. For example, this cannot work for any application which uses dlopen() and packages its loadable modules independently of the invoking binary. This is a problem for very high profile software such as:

* Apache, and just about any other application server with a plugin interface * Perl, Python, Ruby, Java, and just about every other interpreted/VM oriented language

Furthermore, there exists other intractable conflicts in complex library dependency interactions. For example:

bin/A needs SONAME=B.so.1 needs SONAME=C.so.1 (v1)

bin/B needs SONAME=D.so.1 which needs both SONAME=C.so.1 (v2, incompatible) and SONAME=B.so.1

The dependency relationship of every shared object in this chain are not static, and you cannot modify the invoking binaries (where RPATH must be set) every time an intermediate shared object changes its linking requirements. You also cannot guarantee ordering when a single node depends on multiple libraries -- the order of evaluation is non-deterministic. You might say this is a corner case, but you would be surprised when you try at scale. This is not a corner case.

Seriously, I understand everything about this. Can't be done. You feel free to try it out and see that it doesn't.

Just to clarify: the purely functional part is how the package are managed. Nix itself is mostly written in C++, and I'm not sure, but probably not in a purely functional style.

Reply on news.ycombinator.com