Hacker News Reader

Comment by speed_spread | Hacker News Reader

One could develop a whole package manager based on this. Add some standard build profiles to something like Gentoo's portage to maximize cache hits and you could have the best of both worlds: a fast source-based distribution. Specifying a custom build profile for a package would just mean it would take more time to install but would otherwise be transparent.

One thing missing from this is a trust framework where cache writers would sign their compiled code. There could also be a verification layer.

View on HN

Warpforge -- project website: http://warpforge.io / https://github.com/warptools/warpforge -- is a project I work on that's heading a bit in this direction. Hashes in + build instruction = hashes out. It's really a powerful primitive indeed.

Building big sharable package systems is indeed a second half of the problem. Figuring out how to make the system have the hashy goodness AND be usefully collaborative with async swarms of people working independently and yet sharing work: tricky. We're trying to do it, though!

We have our first few packages now published, here: https://catalog.warpsys.org/

You can also see _how_ we build things, because we publish the full rebuild instructions with each release: for example, here's how we packaged our bash: https://catalog.warpsys.org/warpsys.org/bash/_replays/zM5K3V...

I'm in #warpforge on matrix with some collaborators if anyone's interested in joining us for a chat and some hacking :)

I wrote a little bit about this a couple of years ago: https://blog.williammanley.net/2020/05/25/unlock-software-fr...

gittup[1] implements part of this idea, but without the distributed bit. bazel[2] implements the distributed bit (minus trust), but not the distro bit. What's really lacking is momentum around the idea to get a sufficient number of people behind it.

[1]: https://github.com/gittup/gittup [2]: https://bazel.build/

NixOS uses a cache that works like this; it's source-based like Gentoo but for packages in cache it will download the cache rather than rebuilding. (it will rebuild if you configure the package in such a way to invalidate the cache, or install a package not in cache)

Besides the global cache for the whole distro, you can also set up caches for other software. For example, if you build your projects with Nix, you can have a cache for your projects (so that new contributors won't need to recompile everything from scratch). That's the premise behind https://www.cachix.org/

The only difference is that Nix caches aren't fine grained like ccache and sccache

Gentoo supports building and installing binary packages from portage: You can compile once and then distribute the resulting package to other computers where they can be installed by Portage. Portage even verifies that the USE flags are the same!

Nix is also almost there - it happily uses binaries if they match the exact build inputs but compiles if no binaries match - but it's not nearly as fine grained (you'd need something like 1 package per object file).

And the saddest part is that Nix could be exactly there and solve another one of their problems at the same time: content-addressed outputs.

It's called nix

Reply on news.ycombinator.com