Flakes – Proposed mechanism to package Nix expressions into composable entities | Hacker News Reader

Flakes – Proposed mechanism to package Nix expressions into composable entities

github.com

164 pointsbased252 comments

This is really interesting. Right now you can't really compose nix expressions from several different sources. There are ways of doing it, but they all involve one hack or another that limits the usefulness of the result. This is why nixpkgs is a huge monolithic repository.

Flakes would allow package definitions from different sources play well with each other. This means a bunch of things:

1) Nixpkgs could be broken up into several package sets for better scalability. Eg, haskelPackages and pythonPackages could be broken out into separate repositories.

2) Package authors could provide package definitions along with the source.

3) Companies could have a flake for internal packages without having to patch or overlay nixpkgs.

4) Alternatives to nixpkgs can get started without having to supply all the packages that a user might need.

5) You could even create libraries of nix functions that supply no packages at all, but get used by other flakes to define packages. For example, the machinery for composing a python environment.

> 1) Nixpkgs could be broken up into several package sets for better scalability. Eg, haskelPackages and pythonPackages could be broken out into separate repositories.

While this could happen, it's exactly what I hope won't happen, because in my opinion the idea that haskell or python packages are fundamentally different and can live in their own world is a fallacy. I spend quite a bit of my time fixing python packages on nixpkgs master and the number of times the fix actually involves making a tweak to a non-python library means I would forever have to be playing games synchronizing repositories. Also, gone would be the ability to make atomic changes across these boundaries.

Viva la monorepo.

> Viva la monorepo.

Absolutely. Look at edolstra take on this though https://github.com/NixOS/rfcs/pull/49#discussion_r305450668

This is why I'm scared of this change.

Me too. I think (hope) that the NixOS community won't end up doing all the work it would take to break up the existing monorepo; instead they'll just add a few small additional repos.

Nixpkgs as a monorepo is great if all you want to do is install FOSS packages on your NixOS linux box. That's probably the biggest use case for nixpkgs, so it's important, but it's not the only use case.

If pythonPackages were broken out into a separate repository (which I'm not advocating, it was just an example), you'd fix a python package by importing the upstream library from the nixpkgs flake, tweaking it locally and then using the local version as buildInput to the python package. Sometimes it would make sense to submit the fix upstream, and sometimes not.

At the same time, we really have run into scaling issues—automatically importing all of Stackage into nixpkgs was killing us for a while there.

Regardless of whether nixpkgs remains a monorepo or not, I think 2) is actually the most exciting thing about the flakes proposal. Package authors being unable to supply nix packaging within the package repository is a big impediment to the adoption of nix.

I’d love for a way to write nix expressions that easily lead to reproducible builds. Of course this is possible right now next, but requires substantial legwork to pin Nixpkgs version or add SHA for every tarball. Especially now as the machine learning story for NixOS is becoming strong, it would be awesome to have fully reproducible model training using such a system. This highlights a difference from docker or nvidia-docker, which do not control driver version.

Fortunately, `builtins.fetchTarball` makes this easier. I do this in my `nixcfg` repo so that I can build my entire machine config (patches and enabling Iris in Mesa, and all) on a stupidly-cheap Packet.net VM: https://github.com/colemickens/nixcfg/blob/master/default.ni....

If you follow the rabbit-hole: `default.nix` -> `lib.nix`, etc, you can see that I pin nixpkgs, have an update script that updates the nixpkgs revs I build against, it supports building against a custom, local `~/code/nixpkgs` if it exists, and I have my machine config abstracted out to where I can build a "GNOME instance of my machine" or by default, my machine with Sway and sway related packages installed. Much of this should be easier with flakes, as I understand it.

It's not the cleanest config, the README needs love, but maybe it can be inspiration in the meantime. :) My latest trick was figuring out how to get the new Mesa Intel Iris Gallium driver enabled without rebuilding the world, and I extracted it to what I call a "mixin" that anyone can copy and just use: https://github.com/colemickens/nixcfg/blob/master/modules/mi...

(And technically I still use mozilla/nixpkgs-mozilla to pull the latest Firefox Nightly which is impure and thus not always perfectly, perfectly reproducable. I do however pin the overlays themselves also - like my nixpkgs-wayland overlay that packages HEAD versions of Sway and other Wayland related tools!)

This is interesting. Sounds like you do this on the system level? I’m interested in doing this on a shell or package level eg I’d love to have system on bleeding edge but be able to jump back to same binary blobs associated with a particular revision of software

`nix-env` can be coerced into pinning a package at a specific revision, effectively ignoring channel updates. However, yes, I do all of my nixos package management in my system configuration and try to avoid `nix-env` at all costs as it leads to drift. However, in at least one case, this is largely un-avoidable (in the case of OBS Plugins, there's no NixOS infra to make sure that 'wlrobs' is available at a well-known spot for OBS to pick-up. We'd need to make a plugin-aware wrapper, or `programs.obs` module probably. This is because nix-env does some additional symlinking that isn't done for system-installed packages).

As colemickens mentioned, fetchTarball works pretty well. I have an expression that fetches a specific version of nixpkgs, imports it, and instantiates it with an overlay. That lets me override certain packages to customize them or use a different version than nixpkgs provides, and also to add my own packages.

I periodically update the version of nixpkgs it uses, but since it's pinned, I get exactly the same outputs when building.

Ah do you use fetchTarball for nixpkgs? Although maybe I’m not understanding how you’re using it, could you share an example? Interested in your workflow!

Yes, I use fetchTarball to get the pinned version of nixpgs. Like this:

  let
    nixpkgs = builtins.fetchTarball {
      name = "nixpkgs-upstream";
      url = https://github.com/NixOS/nixpkgs/archive/a835adc10cb813d214a9069361d94a2a3f8eb3a5.tar.gz;
      sha256 = "0h4lacvqmk356ihc7gnb44dni6m5qza23vlgl6w6jdhr9pjcmdcm";
    };

    overlay = self: super: {
      # customizations
    };

  in import nixpkgs { overlays = [ overlay ]; }

This isn't even simplified. I just cut out the contents of the overlay. For more on how overlays work take a look at these references; they were really helpful to get me started.

https://nixos.org/nixpkgs/manual/#chap-overlays

https://blog.flyingcircus.io/2017/11/07/nixos-the-dos-and-do...

Is model training reproducible with identical software? I was under the impression that a number of CUDA expressions were non-deterministic, especially in the least-significant bits. And that's if you're running on the same hardware models, there's more variation between models.

It’s largely but not fully deterministic eg backward pass is not deterministic: https://stackoverflow.com/questions/50744565/how-to-handle-n...

That said, it’s hard enough to even run an old model 2 years later let alone reproduce to significant digits. I’d be happy enough to solve the former and get “close enough” training results

While Nix's concepts are awesome, the implementation doesn't look so nice. Their "language" for writing Nix files [1] doesn't look attractive. Doesn't make me want to write a package description.

Of course, it is better than languages with lots of brackets, but still not good enough.

[1] https://github.com/edolstra/nixpkgs/blob/release-19.03/flake...

But since when a language need to looks attractive?

First of all, "attractive" is simply your way to say "familiar to me".

Then, a language need to be just powerful enough to express it's semantic and no more.

This critique is quite hollow and noisy.

Even after writing substantial number of nix scripts and taking some weeks of break (because my job is not writing nix scripts 24/7), I struggle to remember all of that structure inheritance magic and have to reread the docs. Not a good sign, usually happens with me with unnecessarily overengineered things.

What do you mean by "structure inheritance magic"? I can't think of anything I'd term "magic" in the Nix language.

Nixpkgs itself does a lot of funky things, especially around overrides and whatnot, but those are features of Nixpkgs, not of Nix the language.

> Not a good sign, usually happens with me with unnecessarily overengineered things.

Haha yes, I mean clearly this means it's Nix's fault. Has anyone ever considered using you as some kind of "overengineering divining rod" where they can just wave something in front of your nose and go back to the drawing board if it doesn't pass?

Sounds like you wouldn’t like GuixSD either.

Well you might be right, but I find Nix and NixOS to be great. Honestly the language isn’t the issue, it’s just that it isn’t nearly as well supported as more popular distros like Ubuntu, so for example my VMWare Workstation license has been sitting dormant for a while now... (Which is mostly OK. I’ve been using libvirt for everything and pleasantly surprised at what it can do.)

The language takes some getting used to, and I am not really 100% sold on pure functional, but it works and my systems are much more reliable now.

At least that language does not require quotes around almost every sequence of letters like JSON does.

(Yes, I know, JSON is not a language, but people do sometimes use it for things approximately as long and as complicated as a Nix expression.)

cough YAML cough

Oh, man, let's not go there. It's a silly place.

Somehow, right?

But why isn't there an ML with the good parts of YAML, but without the bad parts?

YAML (and TOML and even XML when it's being used in a tag-centric way instead of a text-centric way) are not markup languages. A markup language is one that allows you to assign meaningful annotations to regions within a chunk of textual content (e.g. "these three words are underlined", "this relates to footnote 3", etc.). YAML and TOML have both changed the meanings of their acronyms to try and distance themselves from their original misnamings as "markup languages".

Yep. Maven (ugh) even has switched off entity expansion, a basic feature of XML/SGML, in favour of their own ad-hoc ${property} syntax.

Anyway, the GP's question is a good one. Where are the configuration/serialization languages that import the only good parts of YAML?

AFAIK, one can not subset YAML and get something coherent. But one can start from scratch and use its concepts. I imagine the languages are not there just because nobody got the time to do them.

For basic package descriptions, the language is not terribly different from JSON. Here's a definition of a Python application I'm working on:

    { pkgs ? import <nixpkgs> {}}:
    
    with pkgs;
    
    python3Packages.buildPythonApplication rec {
      name = "quicktill";
    
      src = ./.;
    
      doCheck = false;
    
      propagatedBuildInputs = with python3Packages; [
        dateutil pyyaml
        sqlalchemy psycopg2
        pycups reportlab
        requests_oauthlib
        pygobject3
      ];
    
      nativeBuildInputs = [ wrapGAppsHook ];
      buildInputs = [ gobjectIntrospection gtk3 ];
    }

The only big things here are the fact that this is defined as a function that takes the package repository as an argument to access dependencies (the first line), and the "with" expressions that allow me to skip having to type "pkgs." and "pkgs.python3Packages." for every package I depend on.

That looks completely inscrutable to me. Like a sequence of magic words. I don't see the connection with JSON at all.

I'm not really sure what you're finding inscrutable. I'll tear out the boilerplate that explains that this file is a function and returns a Python application (which is documented in the nixpkgs manual and can just be copy-pasted if you really don't want to figure out what it does):

    rec {
      name = "quicktill";
    
      src = ./.;
    
      doCheck = false;
    
      propagatedBuildInputs = with python3Packages; [
        dateutil pyyaml
        sqlalchemy psycopg2
        pycups reportlab
        requests_oauthlib
        pygobject3
      ];
    
      nativeBuildInputs = [ wrapGAppsHook ];
      buildInputs = [ gobjectIntrospection gtk3 ];
    }

Is that any better? It looks an awful lot like some dialect of something JSON-like to me - there's some different symbols used and array values are separated with spaces rather than commas, but aside from that it's just a record of key-value pairs.

From there, the first line in the original example says "the following expression is a function that is called with this parameter", and the parameter is an object containing a key called "pkgs", which is defaulted to the result of "import <nixpkgs> {}". This is explained in more detail in a handful of lines in the language manual.

Functions are called just by referencing the function and then the arguments separated with spaces, like Ruby or Haskell - so python3Packages.buildPythonApplication is a function that is called with the object above.

Finally, "with foo;" just means "every key in foo can be accessed without typing foo. in the next expression".

The result is basically: this file defines a function which takes an object containing the package repository as the "pkgs" key, and returns the result of calling pkgs.python3Packages.buildPythonApplication with the object defined in the rest of the file.

It's a lot more derived from the syntax of functional languages than C-like ones, but that's not necessarily a bad thing. Also: this is basically the entire language. There's a couple of different string syntaxes and some stuff for merging objects together, and also a mechanism for temporarily naming things that aren't part of an object, but in general... there's not a lot more to it. Most of the complexity is in the domain itself - what's the difference between a propagated, native and regular build input, how do you manage plugins and options for a package, etc.

What's your preferred packaging DSL?

I mostly deal with openembedded at work and I have few positive things to say about it. I'm sure Nix is much faster and nicer. Frankly it's hard to imagine that the opposite could be true.

But your example looks nothing like Json.

I'm not sure what to say. Replace the = signs with colons, the semicolons with commas, stick commas between array elements and stick everything between quotes and you basically have json.

That looks like a Turing-complete DSL. Not like JSON at all. Do we need yet another Turing-complete language for writing package metadata?

The example lacks functions / flow-control statements (ok, apart from "import if not defined", but it's more of a default value than a branch) / references to other elements. It's pretty much key-value dictionary. Not sure how you got an impression of Turing-complete DSL from that, since it would require at least one of them.

The Nix expression language has branches and functions (including recursive functions). It is Turing complete, and it probably is necessary for its charter.

OP said it "looks like" a Turing complete DSL just from the example. Regardless of what extra functionality the actually is, the examples didn't suggest it.

I guess I was giving the benefit of the doubt, since that's such an irrelevant nit to pick...

It's not "package metadata", it's the complete instructions for building an entire operating system in an automated way.

The complicated bits are the import notations (what do the different variations mean and when to use them?), the recursive declarations (and knowing when to use them). Also, since expressions are often functions and the nixpkgs people don’t believe in types or docs, it’s difficult to track down the type info for each argument (what properties does it have?). You just end up grepping blindly through nixpkgs to find the callsite and even then the argument is often the result of a function call, so you have to find the function and it’s definition and look at the return and hope that it is not “return someOtherFunc()” lest you have to track down yet another function (also, odds are “someOtherFunc” is not the name of the function, but rather the name of a parameter into which a function was passed, so you’re back to blindly grepping nixpkgs).

Nix is great in concept but the language or idioms make it really difficult to use.

By the way, there is a list of dependencies, but no version numbers. Is that ok? Even if in theory packages should be backward-compatible, it doesn't work in practice.

Regarding syntax, I would prefer something like this:

DEPENDS_ON python 3.5.1, python/dateutil 1.2.3, ...

BUILD_TOOL bash 1.2.3

BUILD_SCRIPT ./build.sh

This works fine in Nix because those package names are the names of derivations that themselves pin a given version of said packages.

In practice, the default set is expected to be somewhat internally consistent, so that you can just use the package names to refer to the "most recent consistent set". But if need arises, you can also override individual packages to pin them at a different version than the upstream-provided one.

I ran Nix for about six months. The language was one of the factors that really bugged me at the time. That and the hoops you had to jump through to do basic things like installing npm packages...

The nix language is a bit hinky; unfortunately nothing available even approaches it in terms of capabilities.

There are various problems with nix and the pragmatics of the language (error reporting combined with lack of static type checking, inconsistent naming conventions, evaluation speed, tooling, etc.) – but the core language itself is, IMO, actually extremely well designed. I can think of very few good external DSLs (postscript is the only one that comes to mind right now) – but I'd definitely place nix among this list.

This proposal makes me scared for my bet on going full with NixOS.

To me it seems like a really complicated technical solution to a social problem(the monorepo approach doesn't scale).

If more people are needed to maintain the nixpkgs repo why instead not focus on ways to increase the community size.

Missing docs have been a problem for NixOS for a long time, most awesome features are lost on some proficient users tweets. Having better docs would do wonders for the community.

I think if NixOS wants to copy some of Rust strengths, quality docs should be its first priority.

> the monorepo approach doesn't scale

Citation needed.

That's mentioned by edolstra in a comment on the proposal.

https://github.com/NixOS/rfcs/pull/49#discussion_r305450668

I know who the author of that comment is, but it is not a universally held opinion among Nix users.

Looks good, but makes me worry that target sponsored this. Are RFCs normally cooperate sponsored?

I had the opposite reaction--it's great for NixOS to be gaining corporate support!

Target is also behind lorri: https://github.com/target/lorri/

I view corporate interest as a good sign for the continued health of the ecosystem. As long as they don't "take over" the project there's no harm. And in particular, publishing an RFC like this shows that they're doing things the right way and cooperating with the community.

It will replace nix channels? Looks like a big change!

Reply on news.ycombinator.com