Hacker News Reader

This is one of the most interesting Nix tools I've seen in awhile!

Nix is a fantastic tool for building isolated work artifacts, but not the greatest for actually scheduling work across a fleet of machines.

This combines both worlds, I hope this succeeds and gains adoption!

Thank you! We appreciate it.

This looks a lot like Nixery[1]. Can you explain how they differ?

[1] https://nixery.dev/

Hi flurie, Nixery exposes an API (in the form of an Docker registry) to dynamically build Nix-based images, but still tar & compresses Nix packages into layer tarballs. Since the image spec has a limit of 128 layers (due to overlayfs), a heuristic is used to put popular packages together. However, in practice there is still a large amount of duplication between images that share the same packages due to this heuristic based strategy. You also may deploy the same packages outside of containers but have to duplicate that same data (in a slightly different format) on a Docker Registry.

With `pkgs.nix-snapshotter.buildImage`, containerd natively understands Nix packages, so everything is pulled at package granularity without the layer limit. This means all container content is either already in your host nix store or fetched from your Nix binary cache. Think delta image pulls, which isn't possible with regular images since one bit difference will change the layer hash and thus duplicate.

That said, nothing stops Nixery from building nix-snapshotter images. Then we'll have an Docker Registry that dynamically builds native Nix images.

So if the package isn't in cache, then suddenly k8s node is building it from source?

If you are pushing nix-snapshotter images to a Docker Registry, then it can only either use what's in your host nix store, or fetch from a binary cache.

The same is true if you are using the special image reference like `nix:0/nix/store/f8b1hia3hcqwa5d46anzy3cszi3s6ybk-nix-image-redis.tar`, it just means that instead of fetching the image manifest from a Registry, you are using the Nix protocols.

You will only build from source dynamically (and a mix of caching) if you are doing `kubectl apply -f ${podSpec}` as a full deployment Nix expression. See the README's asciinema and this file as an example: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/e...

I think your usage example will be easier to understand, if it explains where nix:0 is setup as CRI image-service. Maybe I glanced over it.

Awesome project!

Hi Alex, I’ve updated the README with a bit more information.

0 is actually an unbindable port, so nix:0 is just an arbitrary string that is unlikely to have conflicts. The Kubelet needs to be configured with —image-service-endpoint to use the nix-snapshotter gRPC socket to handle the “PullImage” RPC. There we can take over resolving the image reference when it’s prefixed with nix:0.

Oh, I see. Thank you for explaining. I misunderstood how this works.

Super interesting. I can clearly see how this improves the caching story compared to vanilla Kubernetes+containerd, and therefore the potential this has to improve autoscaling performance.

Of course, it understandably requires the host VM to be NixOS, which means my Kubernetes cluster needs to be running NixOS, which means I would need to abandon the ease afforded by EKS/GKE... which is a tall ask. It would be great if there was a way to more easily add this into popular managed Kubernetes distributions, to play around more with running "Nix images", simply by adding nodeSelector and/or node affinity to the Kubernetes manifests.

I don't think this is true, I think it only requires the host system to have nix installed.

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service.

I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later.

In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

I would definitely appreciate seeing some docs with quick pointers on setting that up for EKS at least, if not also GKE!

I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups.

I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?

> I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time

I got the impression that it uses the node's nix daemon.

From the project readme's FAQ https://github.com/pdtpartners/nix-snapshotter#faq

> What's the difference between this and a nix-in-docker?

> If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level.

> nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.

Yeah, I think that's the "main" idea of it, but it was also mentioned that this can work in GKE/EKS if you start a new daemon each time.

The EKS/GKE integration involves modifying the host that the kubelet lives and adding nix-snapshotter as a sibling host service.

I didn’t mean running nix-snapshotter as a Kubernetes resource because then there’s a chicken & egg problem. Kubernetes needs nix-snapshotter image service to resolve the nix-snapshotter image.

I'm not sure exactly how the communication of components works there, but in k8s can this be deployed using daemonset rather host service?

So that means that it needs the host to run nix, right? I suppose that could be useful, but if I'm running nix-built images as containers it's usually because I want to run somewhere that's not "nix native".

If you have a nix environment and want to just kubernetes for orchestration, this is exactly what you want!

That's fair, adding another service is definitely a downside. If you are running nix-built images though, the underlying data still needs to land somewhere on disk. Nix-snapshotter lets you store less data and download less data due to its package granularity.

We already have rootless containerd and nix-snapshotter. And I believe there is work underway to run rootless nix (and rootless k3s too). You may be able to run the whole thing unprivileged one day.

Even at the cost of the k8s nodes having to be "special", this is still fabulously useful running Nix-native workloads on stuff like GitLab or Jenkins workers.

what is performance of image building with this? can it build isolation container without image build https://iximiuz.com/en/posts/you-dont-need-an-image-to-run-a... ?

If you are comparing non-Nix image build performance, Nix is comparable to BuildKit in that it’s able to parallelize a build graph and cache intermediary outputs. Nix doesn’t require a Dockerfile to build, and constructs the layers directly like in the article.

In terms of image size, since it’s dependencies are explicit (all the way down to glibc and lower), the Nix image is comparable to a scratch image with a single statically compiled binary.

Comparing to the existing Nix built images, we are moving image build performance from O(n) (n is number of Nix packages) to ~O(1) because the build process becomes just constructing a JSON referring to already built Nix packages.

I’m guessing it also makes building variations of images much quicker because you only need to build the “difference” between the images?

Yup. If you also deploy the Nix packages to bare-metal, then creating containers is at almost zero cost because the container image component is just JSON.

Finally, something cool in this space that isn't a naked VC grift. Gl Edgar and Robbie.

I am still sceptical. What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage?

I dont see a killer selling point, just some middle ware with extra containerization features which also could be pulled into nixpkgs.

But i have to admit, in my perception bubble VC gets more desperate and to them, the search metrices for nix must be shining like a christmas tree. I am certainly biased negatively.

> What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage?

This is addressed in the readme FAQ. https://github.com/pdtpartners/nix-snapshotter#faq

If you're running Nix-based images on Kubernetes, seems like this tool has reduced overhead / storage compared to if you build an OCI/Docker image.

What's the advantage over me putting my Nix config in e.g. a Dockerfile?

To answer that, I think you’d first need to define what “putting my Nix config in e.g. a Dockerfile” means.

Do you mean using a Dockerfile to run a Nix package build, embedding the resulting files (and transitive dependencies) in the image file? If so, this objectively superior in several ways: 1) only one copy of any package exists on disk: the one in the Nix store, instead of multiple images having independent copies. 2) Related to this de-duplication, it will be faster to go from pristine host (virtual machine, dev machine, whatever) to having containers up and running, as you no longer have bulky images to transfer over the network. 3) I haven’t looked at the finer details, but I’m pretty sure this means that the host can build the required packages once, meaning better caching — instead of multiple Dockerfiles rebuilding the same packages, the host builds once and all dependent containers get use the host’s Nix store, again without any copying of files involved.

Edit:

From hinshun’s comment above, point #3 is incorrect — either the host’s Nix store would need a copy already present, or the package would need to be available in a remote store/cache. Even with that correction, you still avoid redundant package builds — either the package is available remotely/locally, or the image fetch fails — so things are still only ever built once :)

It can leverage de-duplication in your host nix store (as a sibling service to your container runtime), as opposed to having a containerized nix store each fetching its necessary packages.

Will you be at NixCon? Seems like good timing to post this!

Yes! Our talk wasn’t accepted but Robbie and I, and a few others from my team will be there. Come say Hi!

Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd