hinshun

No worries. I think Make vs Ninja is a whole another discussion but how it relates to this project is that Ninja has less features than Make (e.g. no globs, etc) which makes it attractive as a simple build graph format.

Nix provides the toolchain and dependency management (like where boost is from), as well as ability to remotely execute it which makes it attractive to scale out large builds like a `-j 999`. Nix also lets you do things like patching boost and then recompiling both boost and the downstreams (incrementally and with early cut-off if using nix-ninja) all in one build graph.

All in all, probably not useful if you're not already needing features from Nix. But if you are, this should speed up your builds by a significant amount.

We're still missing this feature to fully incrementally compile nix: https://github.com/pdtpartners/nix-ninja/issues/19

Since there's a target that depends on a generated source file for Nix's bison parser. But other targets incrementally compile comparably with regular ninja. So far we observed that Nix's sandboxing overhead is neglible.

I'm not sure, but we're in contact with some core Nix folks to see what we can do help move it along.

It made sense internally and we also thought it'll be compelling for people interested in Nix since Nix itself uses meson and outputs ninja.

What other formats do you think are worth targeting?

Happy to help. For EKS there’s a blog (https://blog.realvarez.com/using-estargz-to-reduce-container...) that goes into using stargz-snapshotter which will be same initial setup, but you’ll need to install nix as well in the “managed node group”.

I’m not sure what you mean by modifying your own installation. Like running k8s on NixOS and then using a nix-snapshotter based DaemonSet to modify k8s on the host? At first glance it seems like vanilla k8s can do this already, nix-snapshotter just makes it more efficient / binary matching.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

Author of nix-snapshotter here.

Yes, one of the main downsides of Docker images using Nix is the 128 layer limit. It means we have to use a heuristic to combine packages into the same layer and losing Nix’s package granularity. When building containers with Nix packages already on a Nix binary cache you also have to transform the Nix packages into layer tarballs effectively doubling the storage requirements.

Nix-snapshotter brings native understanding of Nix packages to the container ecosystem so the runtime prepares the container root filesystem directly from the Nix store. This means docker pull == Nix substitution and also at Nix package granularity. It goes a bit further with Kubernetes integration that you can read about in the repo.

Let me know if you have any other questions!

The EKS/GKE integration involves modifying the host that the kubelet lives and adding nix-snapshotter as a sibling host service.

I didn’t mean running nix-snapshotter as a Kubernetes resource because then there’s a chicken & egg problem. Kubernetes needs nix-snapshotter image service to resolve the nix-snapshotter image.

Hi Alex, I’ve updated the README with a bit more information.

0 is actually an unbindable port, so nix:0 is just an arbitrary string that is unlikely to have conflicts. The Kubelet needs to be configured with —image-service-endpoint to use the nix-snapshotter gRPC socket to handle the “PullImage” RPC. There we can take over resolving the image reference when it’s prefixed with nix:0.

Yes! Our talk wasn’t accepted but Robbie and I, and a few others from my team will be there. Come say Hi!

Yup. If you also deploy the Nix packages to bare-metal, then creating containers is at almost zero cost because the container image component is just JSON.

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service.

I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later.

In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

Thank you! We appreciate it.

If you are comparing non-Nix image build performance, Nix is comparable to BuildKit in that it’s able to parallelize a build graph and cache intermediary outputs. Nix doesn’t require a Dockerfile to build, and constructs the layers directly like in the article.

In terms of image size, since it’s dependencies are explicit (all the way down to glibc and lower), the Nix image is comparable to a scratch image with a single statically compiled binary.

Comparing to the existing Nix built images, we are moving image build performance from O(n) (n is number of Nix packages) to ~O(1) because the build process becomes just constructing a JSON referring to already built Nix packages.

That's fair, adding another service is definitely a downside. If you are running nix-built images though, the underlying data still needs to land somewhere on disk. Nix-snapshotter lets you store less data and download less data due to its package granularity.

We already have rootless containerd and nix-snapshotter. And I believe there is work underway to run rootless nix (and rootless k3s too). You may be able to run the whole thing unprivileged one day.

It can leverage de-duplication in your host nix store (as a sibling service to your container runtime), as opposed to having a containerized nix store each fetching its necessary packages.

If you are pushing nix-snapshotter images to a Docker Registry, then it can only either use what's in your host nix store, or fetch from a binary cache.

The same is true if you are using the special image reference like `nix:0/nix/store/f8b1hia3hcqwa5d46anzy3cszi3s6ybk-nix-image-redis.tar`, it just means that instead of fetching the image manifest from a Registry, you are using the Nix protocols.

You will only build from source dynamically (and a mix of caching) if you are doing `kubectl apply -f ${podSpec}` as a full deployment Nix expression. See the README's asciinema and this file as an example: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/e...

Hi flurie, Nixery exposes an API (in the form of an Docker registry) to dynamically build Nix-based images, but still tar & compresses Nix packages into layer tarballs. Since the image spec has a limit of 128 layers (due to overlayfs), a heuristic is used to put popular packages together. However, in practice there is still a large amount of duplication between images that share the same packages due to this heuristic based strategy. You also may deploy the same packages outside of containers but have to duplicate that same data (in a slightly different format) on a Docker Registry.

With `pkgs.nix-snapshotter.buildImage`, containerd natively understands Nix packages, so everything is pulled at package granularity without the layer limit. This means all container content is either already in your host nix store or fetched from your Nix binary cache. Think delta image pulls, which isn't possible with regular images since one bit difference will change the layer hash and thus duplicate.

That said, nothing stops Nixery from building nix-snapshotter images. Then we'll have an Docker Registry that dynamically builds native Nix images.