Hacker News Reader

It is great to hear about your work in the space!

Several years ago I looked at implementing a custom component for k8s which would exchange nix store paths instead of containers, substitute, and bind mount them in at run time. It was an interesting experiment, but was Quite Difficult to pull off for someone who wasn't already familiar with k8s.

I've seen some projects similar to what you're describing though: the lightning-fast substitutions. It was incredible! They had the benefit of a fabulously fat network connection, though, and I'm not sure the experience translates very well. We will see!

Thanks! The domain is robotics, and we gave a brief conference talk last fall about our experiences so far, if you're interested— although it's mostly directed at a Nix-unaware audience, you can get a glimpse of our workflow that's heavily oriented around hourly builds delivered as Nix flake tags: https://vimeo.com/767139940

Definitely there's an obviousness to the concept of "magic" Nix stores in various spaces, and I know the tvix project seeks to realize some of this as well— reusing OCI tools to supply the build sandbox, leveraging existing container orchestrators for job management and queuing, all those goodies. So I'm excited to keep watching the space.

> But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command.

Mostly a digression since it's not Nix, but I've wondered a bit about this sort of thing when building AUR packages on Arch Linux. The very last step of the builder is to compress the package, which in the vast majority of my uses is followed immediately by installing it, which of course decompresses the package. I've wondered why there isn't some (non-default) option to say "I don't need to keep the package itself around; just install it as soon as its built". I'm sure for my specific use case there's a simpler solution, but I've always wondered if there's a hacky way to get around things more generally by making a tool that can mimic the expected compression API but then creates a "fake" compressed artifact that no-ops (or maybe puts in a valid header followed by non-compressed data) and then injects that implementation into the PATH. You'd be able to invoke it with something like `fakecompress --zstd makepkg -si`, and it would invoke `makepkg -si` with the no-op zstd implementation.

Ironically, Python has moved in the other direction, with now requiring a wheel be built before installing it, when wheel itself was originally an optional bolt-on to setuptools.

I know you wanted to make a general point but in Arch you can just change PKGEXT='.pkg.tar' in /etc/makepkg.conf and the compression stage is skipped.

https://wiki.archlinux.org/title/makepkg

Tar would also be the answer to how to do fake compression (keeping files together and in order in one file) IF you can choose the "compression" library/tool.

Interesting! I wasn't aware of that, so I'll check it out

Any pointer to flakehub?

I think it's mostly just a prototype: https://github.com/elohmeier/flakehub