Hacker News Reader

Comment by 8organicbits | Hacker News Reader

When doing reproducible builds with Nix, how are people dealing with build artifacts that can't be downloaded? I see that I can do a fetchurl with a specific URL and hash, but this obviously fails if the content is missing.

On Debian pinning is a little strange. You can try `apt install foo=1.2.3` and it will work until the next versions is released. Debian removes old versions from the mirrors. However they operate a snapshot archive with all the old package versions, so reproducible builds can use that and should continue to work over many years.

Does Nix have a way to ensure pinning continues to work over many years?

View on HN

> this obviously fails if the content is missing.

It doesn't, actually - all the fetchers are [fixed-output derivations](https://nixos.org/manual/nix/stable/language/advanced-attrib...) where you specify not only the source, but the hash it is expected to produce. And so the fetched files will be hashed and placed in the nix store under that hash. And therefore subsequent builds using that artifact will use the already-stored artifact and not care if the source is reachable or not. You know that all "external" inputs will be captured by fixed-output derivations, because any derivation that isn't fixed-output is blocked from network access.

For anything built by the hydra CI, this will be retained on central cache.nixos.org. Or you can copy from a store of some machine that has it.

Now, if it's something not on the official cache.nixos.org, and the source is gone, and you've lost (or pruned) your store, you will need to get the file from somewhere and put it back in your store: https://nixos.org/manual/nix/stable/command-ref/nix-store/ad.... Or you just change the source URL (this won't change the derivations depending on it, because the input being transferred is identified by the hash that is to be delivered, not by the URL it comes from).

Looks like cache.nixos.org may be a long term archive (never evicts artifacts) so that is suitable for long-term reproducible builds for nixos things.

If I'm following correctly, this would prevent issues like builds failing due to the leftpad package getting unpublished, since you'd use the cache/archive instead of failing to fetch the missing package.

> and you've lost (or pruned) your store

It's common for academic code, for example, to be built once, published and then shelved. The "you" in "your store" ends up being different people over time.

Apt generally has this issue too, while Debian keeps old packages around (snapshot archive), third parties origins and other apt based systems may not.

I'm going to explore nix more, thank you!

maybe not if you want to build from source and the oriiginal tgz/git goes offline?

For things in nixpkgs, the files will be mirrored on the nix cache. For your own stuff you could set up your own cache or use a service like cachix. There is also a nix option to set up a generic mirror which looks up files by hash, but I'm not sure if it works with all the ways to fetch things: https://nixos.org/manual/nix/stable/command-ref/conf-file#co...

Packages that make it into nixpkgs (and are not ignored by Hydra because e.g. they're non-free) get copies of their fetched sources included in the nixpkgs binary cache. So they're there for "posterity" (there's no such thing as posterity, though I'm sure some wise people are maintaining full personal backups of the nixpkgs binary cache).

Reply on news.ycombinator.com