Building Container Images with Nix | Hacker News Reader

Building Container Images with Nix

thewagner.net

126 pointsdboreham75 comments

If you want to take nixos for a spin in your browser head to https://github.com/gitpod-io/template-nixos and mash the button.

Also https://nix.dev is one of the better learning resources. Over at https://GitHub.com/ghuntley/ghuntley you'll find a workshop I created on Nix.

Nix is really cool for demos like this, but seems to get complicated if you need to grab specific versions of a dependency. At last check, that involved digging through the history of the package/channel, grabbing the hash of the package corresponding to the desired version, and then referencing the package by hash, all with basically no tool support?

For system that is so proud of allowing multiple versions of dependencies to coexist, it seemed surprisingly difficult to actually find and use specific versions of a dependency in, you know, applications.

Flakes, which is currently an experimental feature, is gonna solve the issue. It works similar to cargo and npm, in that it pins the dependencies and creates a lockfile.

As I understand it, flakes don't really solve the problem parent is talking about. The discoverability is still poor. Say I need a python package at version x.x.x. How do I find which nixpkgs commit hash I should use to get that specific version?

I believe you can override the version property of the derivation that builds the package, if its build script is compatible enough between the two versions.

> How do I find which nixpkgs commit hash I should use to get that specific version

This is answered in a sibling comment here: https://news.ycombinator.com/item?id=28243409

The tool does exactly that.

However:

> find and use specific versions of a dependency in, you know, applications.

This is not what nixpkgs currently provides, or aims to provide.

nixpkgs is like the Debian, Gentoo, etc. package repos: In most cases, it provides one version of a package. If you want to use an older one, you have to use older version of all dependent packages, as they were at that time; this in practice means using software with security vulnerabilities that are only fixed in current nixpkgs.

nixpkgs is not currently designed to let you compose your application from libraries taken from different nixpkgs versions. It is insecure (see above) and slow (evaluating more than a few different nixpkgs versions takes a long time).

The current correct way is to use a current nixpkgs version as a base, and then use an override to update the sources of the packages you want to have a different version of explicitly. Or use a tool that does this automatically for your programming ecosystem based on that one's package manager, like `stack2nix` for Haskell, `yarn2nix` for Javascript, and so on. Nixpkgs makes such overrides easy.

What you describe is weird.

It's very common to pin dependencies to specific versions. And upgrade them carefully, also to specific versions.

If I use a dependency at version X.Y, a vulnerability is discovered in it, and fixed in version X.Y.Z, I don't want to "start from base nixpkgs". I want to upgrade that specific dependency to a very specific version.

From how you described it, it's either not possible or extremely cumbersome to do with nix. Which begs the question: what's the point?

> I want to upgrade that specific dependency to a very specific version.

Yes, this is what I am arguing you should do, by overriding its source to that version (which I described as being easy).

You should not try to pull in a version of nixpkgs (whole OS) that has X.Y.Z, and combine it with another version of nixpkgs (whole OS) that has some other library's version A.B.C.

In the same way as you don't try to combine 3 different versions of Debian to get specific versions of 3 different Python libraries.

Concrete example:

Current nixpkgs has libpng-1.3 and libjpeg-4.6. For building your app you desire libpng-1.2.3 and libjpeg-4.5.6.

To get them, you use the current nixpkgs, and override `libpng` and `libjpeg` to the versions you want using e.g. `libpng.override { src = fetchGit ... }`. You do NOT do `libpng = (import <older-nixpkgs-version-1> {})` and `libjpeg = (import <older-nixpkgs-version-2> {})`. Because that way you will also end up with 3 different versions of e.g. `glibc`, which may be incompatible.

I hope this clarifies it!

We need to specify hashes for `libpng-1.3` and `libjpeg-4.6` right? Is there tooling to fill-out them automatically?

How to find out what versions of `libpng` are available to be used in `shell.nix` or `shell.flake` with nix tools?

If your dependency is on GitHub, this tool makes it super easy. https://github.com/seppeljordan/nix-prefetch-github

https://lazamar.co.uk/nix-versions/

The use case you're describing is impossible with other packaging tools, so really the choice is either "nothing" or "write a nix script".

(Tool support would be nice, but nixpkgs has no standard way to specify packages across languages and runtimes, so not possible right now.)

That's a fair point, I don't have the same expectation of other os-level package managers.

There are alternatives in other projects that bridge build system and package management while directly addressing versioning of dependencies e.g. Spack, Conda, ... The first in particular does exactly what I want, so I suppose I should just be happy with it. Still would be nice for Nix to compete in the same space, esp. with the really slick translation from local to containerized environment displayed in here

Nix can do what you want, it just requires a little bit more manual intervention.

There's benefits, too, though - you know the C compilers, build flags, versions and libraries of everything that goes into making a Python library run. It's easy to change any of it, integrate with your own custom libraries or other languages and test the whole contraption in its entirety.

This is whats so powerful and wonderful about it, your end result isn't simply based on the direct dependents, but also includes the build chain for all things.

That's just not that common in other build tools.

Incredible, many many people responded to you, no one gave you a useable answer at all.

There are at least two answers.

The most popular one, which I disagree with, is to find an older nixpkgs hash and just use that. This works of you don't care about having all dependencies downgraded or security updates of anything in nix. There is even this tool to help you do it: https://lazamar.co.uk/nix-versions/

However, I dislike that method a lot.

The correct method is via overrideAttrs possible with an overlay. This post explains that pretty well: https://discourse.nixos.org/t/how-to-override-package-versio...

Interesting, thanks for the pointers.

So AIUI the second, recommended method, it's overwriting the source location for that particular package, and then letting nix build that with an otherwise unchanged nix recipe/script/thing? Am I responsible (at the point of use) to make any other changes required to make that version build successfully? Do dependents of that package get rebuilt too?

An overlay could define a package with the same name as one from a lower level.

IE python = super.python...

It could also completely override the entire definition of something if you wanted.

IE python = super.stdenv.mkDerivation { ... };

But you can also add new names, and not modify the existing definitions.

IE python-for-me = super.python.overrideAttrs (...);

In the last case you can reference your modified copy while other things in nixpkgs can use the one built upstream.

I use Nix to provide consistent build environments (and then use whatever project tooling). This allows for an even simpler approach. I write a shell.nix that gives my dev environment. Developers can use that while developing to get a consistent build environment. I can then set up CI in Gitlab by just applying that environment as a before_script. Something like this can be a complete CI file:

    image: nixos/nix:latest

    before_script:
      - nix-env -f shell.nix -i -A buildInputs

    test_build:
      stage: test
      script:
      - elm make elm-src/Main.elm

If you've got loads of dependencies and/or they need to be built and take a while then this can obviously slow down your CI and you'll want to build the docker image. For simpler projects, it is incredibly quick, simple and robust to set up.

Is it possible to cache the already built dependencies in Github or Gitlab CI if one is using a NixOS image?

I believe so.

https://cachix.org/ provides some off the shelf services.

There's a few blog posts about using AWS S3 as a binary cache so you can roll your own.

Cachix is really awesome.

There's a few blog posts about using AWS S3 as a binary cache so you can roll your own.

Unfortunately, this is really slow due to Nix using Brotli compression at a high compression level. There were some PRs to make the compression method configurable, but the last time I checked, none of them were merged.

It is not really an issue if your outputs are small. But I had a private S3 cache and have some derivations that have large output paths, and it made CI excruciatingly slow.

I think of building the docker image as a 'cache' of the dependencies, but actually something like Cachix would provide a more flexible solution (i.e. helping developers as well as CI).

Docker is a bad way to handle this. Cachix is the correct solution.

Cachix is definitively the easiest (don't waste my time) solution.

If you are on AWS you can also set up S3 bucket.

There are multiple caching mechanisms and you can use them together to provide multiple levels.

The mentioned cachix (you can also create S3 bucket that can also provide similar functionality).

If you set up gitlab runner in NixOS [1], you can follow example configuration which allows to utilize nix local caching.

You can of course also use the normal gitlab caching, you will need to ensure the container has /cache mount.

[1] https://search.nixos.org/options?channel=21.05&from=0&size=5...

Here's the gitlab-runner configuration I mentioned in the post (can't edit anymore)[1]. With this you benefit of local caching that nix does. What this means is that if you have a branch and it compiled, and run test. Then as you merge and no source files changed (which happens most of the time), the build on master is instant.

[1] https://github.com/NixOS/nixpkgs/blob/ce38fecabe49a76fcd08d0...

The problem with Nix still remains it's documention, as this blog illustrates so well.

- MAny links points to a bunch of 404's. - The links that don't 404 go to ubiquitous readme's that have so much information it's hardly a good place to start. - The demo he provided is the easiest of use cases with a single package and static files. How do I actually build a binary? How to accomplish the pip installs he shows with the Docker file? Why not show a really 1:1 example if it's so easy?

And even if I work through all that for this blog, fat chance I'm going to convince my whole team to do so.

The 404s are to the author’s own GitHub repo and are not a reflection on Nix. It looks like they deleted and recreated the repo with the name wagdav.github.com and just didn’t update the links in this post accordingly.

But it is reflective. I've found that sort of thing rampant in it's community, and sometimes it's own docs. It doesn't mean it's bad tech, but will be hard to convince a corporate group to want to use it.

I used this exact idea to do not just containers but cross compiled containers with cached artifacts using cachix, https://github.com/openenergysolutions/openfmb.adapters for a size-able example perhaps

Link 404

Fixed

I found Burke Libbey’s videos really useful to get a good grasp on Nix https://youtube.com/channel/UCSW5DqTyfOI9sUvnFoCjBlQ

I wish he'd open-source runix.

https://gist.github.com/burke/72ca46c80e57a25907a75611ee5eb6...

I think it is too heavily integrated with Shopify that it probably is difficult to make it a general tool.

I created something similar for my needs, although perhaps it is a different, but it was inspired by the configuration file :)

Looks like Burke's runix is about starting multiple services, to create dev environment.

My tool creates an opinionated dev environment (you enter nix-shell or direnv and everything is already set up) that is already integrated with Nix, that means out of the box you can generate nix package, or build a docker image.

Right now it only supports python (poetry) and go which is what I need right now but other languages can be added later.

Not sure if others would find it useful, technically you could just create own shell.nix and default.nix, the problem is that it ends up being a lot of boilerplate and a lot of nix code which scares away people not familiar with Nix.

We're using Bazel + Nix together to build Docker images containing Python apps and it's been working surprisingly well. Bazel builds the final image on top of a base image built by Nix (containing only Python sourced from Nix's repository). All hermetic and no Dockerfiles.

I'm not a huge fan of Nix's expression language, but the system works well.

Wait, what's bazel giving you if you already use nix?

Bazel is where I started. As an ex-Google employee it's familiar, and I prefer its build language. I found Nix when looking for ways to make Python builds reproducible across machines.

Also, I get the impression that a Nix artifact is generally an entire library/package, whereas a Bazel artifact is often just one or a few files which fits well with how I prefer to use the build system. But I'm no Nix expert and that may not always be true.

If Bazel had a better external dependency story (maybe someday [0]) then I'd probably not use Nix. But currently the pair seems to work well.

[0] https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3Zu...

I'm glad you're building containers. Are you using rules_docker or are you using nix to make the containers and just copying files from bazel-out?

The former. See an example I made for the sibling thread.

Nix doesn't have incremental builds. It's kinda all or nothing at the package level.

This explains clearly why you'd want to combine both:

https://www.tweag.io/blog/2018-03-15-bazel-nix/

Is there any kind of writing about it? I'm very interested.

Not sure of any writing that ties all of the pieces together, but I made a small example for you: https://github.com/jvolkman/bazel-nix-example

It's really fascinating. I filed a bug, but no need to respond unless you find it interesting. This is a really cool direction for me, and I really appreciate you putting this up.

It's striking how heavy the WORKSPACE is compared to everything else.

Yeah, I think it's due to the small size of the rest of the example. It wouldn't grow more than a few more lines for a sizeable python app.

We're doing a very similar thing.

Having a hell of a time getting a Nix-built CUDA image to work with EKS GPUs though.

There is data-science channel on Matrix and Discord!

Oh, care to link?

With this stack, do you ever run binaries outside of the containers? Do you need to use the nix command line very much?

I agree that nix is somewhat painful to deal with, but Bazel essentially has no package management story at all. It would be nice to see one of the tools subsume the other, but that doesn't feel like something either project is actively addressing (ie. package management in bazel, or incremental/non-bash-driven builds in nix).

Sure, we can run commands using `bazel run` or build them using `bazel build` and then run the generated binaries from `bazel-bin` directly. In these cases we also use the same Nix-provided python version, but through a different path.

The only time I use the nix CLI tools is when I'm fumbling around trying to make changes to one of our .nix files (we only have a few). Otherwise nix is hidden behind the scenes and I deal only with Bazel.

I am really not convinced of the need of any incremental build system (like Bazel) over Nix excepted if you inherited of a giant mono-repo mess.

Decomposing your dependency tree into small derivation pose no problem with Nix. Mainly due to the fact it solves entirely the ABI problem and the diamond dependency problem.

You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together.

Using a gigantic pile of complexity like Bazel just for that seems kind of an overkill to me. No offense, I am just challenging here :)

I think this view is too narrow; there is definitely need for fine-grained incremental build systems, even outside mono-repos.

If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer.

I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation.

My project also uses non-incremental nix, and an incremental build for our single-package top-level code. We just tolerate the waiting time, but it's not ideal. This works because our project is relatively small and updates to dependencies are not super frequent, so we can spend the time to build ad-hoc hacks to shortcut those builds each time. We find this better than any other currently available way, but it's not ideal.

I'm not convinced that Bazel is the final answer to this (it focused almost entirely on its way to do things, ignoring the package-based world that makes up the open-source eco system). But there's definitely the need for something that addresses both sides, cleanly, and it hasn't been found yet.

> If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer.

I see the point but the '20 hour' build seems to me over-exaggerated.

- Builds in Nix can be parallelized both on single node (multi-core, multi-socket) and multi-node without problems

- Parallel builds 'just works' in Nix with the Nix's sandbox system and the hash-based package identification mechanism. Nothing related to the bug mess that can be a distcc deployment or parallel builds in Bazels.

- Binary cache give a pretty efficient way to reuse any successful past build without problem

- Modifications in "base" packages generally do not happen everyday scenarios.

- Iterations and debugging without majors recompilations can be done with the nix shell directly. Full rebuild are required only for binary shipping.

> I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation.

I had to recompile entire software stack ( > 1500 software ) for exotic archs (U.S supercomputers) in the past. My way was just to burst it on many nodes: compute time is cheap nowadays (and nodes are not missing on supercomputers :) ) .

My time however is not cheap and most incremental builds system are sensible to side-effect (including Bazel) and not as reproducible as they claim to be: Debugging that is in my experience time consuming, much more than a bit a build time.

I do see the interest of incremental builds inter-components, but in the current state of the tooling, it is for me not worth the pain. Specially not the pain of Bazel that has a (very) long list of quirks.

> I am really not convinced of the need of any incremental build system (like Bazel) over Nix [...]

Without smaller compilation units than the default mkDerivation-per-component-style in Nix you end up having _extremely_ long build/test cycles on every change. You could alleviate this by deferring to whatever build system is called in mkDerivation by providing a short-circuiting shell.nix for every subcomponent (eg. a shell.nix to drop you into somewhere where you can `cargo build`). But IMO that's not nearly as nice of an experience as a `nix-build -A some-component` or `bazel build //some/component`, especially when changes span multiple components.

> You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together.

Yes, you can put in the effort to replicate Bazel-style small-compilation-unit language rules in nix. But then your evaluation time will skyrocket, with nix spending dozen of seconds re-evaluating these tiny derivations before even getting to the build - this already happens when evaluating complex derivations like NixOS configuration builds. This also kills your build/test cycle. Plus, you end up polluting your /nix/store with thousands of tiny out-of-date artifacts that then need to be garbage collected (which also becomes slow when the graph grows).

Bazel's build/evaluation model (workspace rules -> targets -> build graph -> action graph -> artifact cache) and the fact that it runs as a server makes it much easier to have snappy rebuilds out of the box without any special consideration for derivation granularity or crutches like shell.nix-per-component. I'm not saying Bazel is perfect, and it's quite difficult to get running in an organization if you don't know it well, but once it's set up correctly it's pretty much the best development workflow I've ever used.

Nix is very good for packaging software that you barely change, eg. for packaging software, as it makes it very easy to hermetically build software that comes with its own build system. But from my experience it's not a great experience for building software you develop, as build times rapidly rise to the point of frustration.

> [...] excepted if you inherited of a giant mono-repo mess.

I'm not sure how to interpret this - both Nix and Bazel shine in monorepo setups (and are IMO a prerequisite to even consider using a monorepo). Even nixpkgs is its own monorepo, and thanks to that is probably the best contributing-to-a-Linux-distribution experience that exists, by a long shot. Are you instead using Nix with flakes accross multiple repositories? Does that alleviate evaluation times?

> ou could alleviate this by deferring to whatever build system is called in mkDerivation by providing a short-circuiting shell.nix for every subcomponent (eg. a shell.nix to drop you into somewhere where you can `cargo build`). But IMO that's not nearly as nice of an experience as a `nix-build -A some-component`

And you describe exactly what we do :)

That's not that bad. The multi-components scenario can also be scripted through some specific nix-shell targets that you configure for a local source usage. It is not perfect but it does the job.

I do consider that if multi-components modifications happen on regular base, it is anyway a sign of a problem: they should probably not be separated components in the first place.

> But then your evaluation time will skyrocket, with nix spending dozen of seconds re-evaluating these tiny derivations before even getting to the build - this already happens when evaluating complex derivations like NixOS configuration builds

We do that by using overlays for our package set ( > 250 packages) over the bigger package set (the entire nixpkgs) which is provided through Nix channels. That allows to leverage the Nix internal cache and keep evaluation under few seconds. Also we do use the nix-daemon.

> Plus, you end up polluting your /nix/store with thousands of tiny out-of-date artifacts that then need to be garbage collected (which also becomes slow when the graph grows).

That is true, but space is cheap :)

> [...] I'm not saying Bazel is perfect, and it's quite difficult to get running in an organization if you don't know it well, but once it's set up correctly it's pretty much the best development workflow I've ever used.

I understand the point of view but I have the opposite experience. Bazel is one of the worst build system I had to experience. The fact he tries to act both as a package manager and as a build system (CMake, make, configure) makes it conflict with almost everything else in the Open Source world which is not made from day 1 for Bazel: This, for me is unacceptable.

I will also pass under silence the long list of quirks it has and the instability of the previous versions.

> Even nixpkgs is its own monorepo, and thanks to that is probably the best contributing-to-a-Linux-distribution experience that exists, by a long shot.

Nix requires only its configuration (Nix's expression) to be in monorepo. At the opposite, Bazel design is centered around the source code being in monorepo. That is a major difference.

One centralize versioning, the other one centralize everything.

> Nix requires only its configuration (Nix's expression) to be in monorepo. At the opposite, Bazel design is centered around the source code being in monorepo. That is a major difference.

Modern Bazel rule sets are more pragmatic than that. Python rules can pull packages using `pip install`, java rules can pull from Maven, rust rules from crates.io, etc.

Much more in-depth article about Nix and Docker, covers how they overlap, contrast and complement one another: https://sandervanderburg.blogspot.com/2020/07/on-using-nix-a...

"some dependencies are only needed to build the image" - it's true if you don't write proper Dockerfiles. You're always free to build an app in a separate image, mark it with `AS build-env`, and then copy the result into an image containing an entrypoint with `COPY --from=build-env`, all within the same dockerfile.

There is also Guix pack -

https://guix.gnu.org/blog/2018/tarballs-the-ultimate-contain...

I'm surprised this post isn't making use of nixery

- https://nixery.dev/

Nixery is a cool idea. The use case for something like Nixery is a more narrow "I want an image with these packages installed". Whereas, with Dockerfiles you've got more flexibility in the image you build.

Hmm not really. That's the toy use case. Nixery builds your expression and gives you a docker container. By default you can specify like "git and tmux" as they are the example toy expressions HOWEVER it becomes a different beast when you use it in monorepo situation see https://GitHub.com/ghuntley/ghuntley and https://code.tvl.fyi/

My understanding is that's still "use Nixery to build docker image with packages installed", but it's a different set of packages being served.

What use cases / dynamics arise from doing that? & what's the value-add from doing that with Nixery compared to just otherwise building docker images with dockerTools in Nixpkgs?

Nixery kinda seems like a toy. Does it have some more serious use?

Come hangout with us https://tvl.fyi/ — the visualisation has some hints :-)

This is a really fascinating idea, and he presents clearly the problems with Dockerfiles and determinism. But, none of the links to his repositories are available anymore. It's a pity, I really wanted to see how you can replace Docker with Nix.

Looks like he renamed (or recreated) the repository:

https://github.com/wagdav/wagdav.github.com/blob/main/flake....

There is something ironic about writing about Nix, which touts immutability as one of the incredible benefits, and having all the links broken in that post.

It’s reproducibility that’s the benefit, not immutability. The fact that the Nix store is immutable is just an artifact of being a content-addressed store, not actually the point of anything.

In the article, here's a dead link in the statement, "Concretely, for the example showed in this section, you can find the complete definition of htmlPages here."

The big problem with both Docker and Nix is that nobody is there to backport security fixes to older versions of libraries and applications.

That's assuming that you start with images. If you start from Nix expressions, you could update your pinned version of the nixpkgs stable branch and rebuild the container image?

Actually that's not true. While docker's reproducibility is based on an image snapshot of a state.

Nix reproducibility is really just a recipe how to build it. Because of that it's not really an issue.

Yes, if you build a docker image through nix you have to regenerate it (which is easy) but that's how docker works.

Reply on news.ycombinator.com