Hacker News Reader

Comment by kazinator | Hacker News Reader

I worked with the internals of this some 16 years ago, maintaining a customized version at Zeugma Systems.

Some changes of mine was reworked by someone and upstreamed:

https://github.com/ccache/ccache/commit/e8354384f67bc733bea5...

If you follow the link to the mailing list posting, you will see I implemented a feature: search through multiple caches.

Say you have a build server which populates a cache. Your developers can point their ccache installations to that. However, ccache wants to write things there. The fix is to have a two-level hierarchy: ccache will look in their local cache first, and then the one from the server. If there is a cache miss, it goes to the local one; the server cache isn't polluted with development objects.

View on HN

One could develop a whole package manager based on this. Add some standard build profiles to something like Gentoo's portage to maximize cache hits and you could have the best of both worlds: a fast source-based distribution. Specifying a custom build profile for a package would just mean it would take more time to install but would otherwise be transparent.

One thing missing from this is a trust framework where cache writers would sign their compiled code. There could also be a verification layer.

I wrote a little bit about this a couple of years ago: https://blog.williammanley.net/2020/05/25/unlock-software-fr...

gittup[1] implements part of this idea, but without the distributed bit. bazel[2] implements the distributed bit (minus trust), but not the distro bit. What's really lacking is momentum around the idea to get a sufficient number of people behind it.

[1]: https://github.com/gittup/gittup [2]: https://bazel.build/

Nix is also almost there - it happily uses binaries if they match the exact build inputs but compiles if no binaries match - but it's not nearly as fine grained (you'd need something like 1 package per object file).

And the saddest part is that Nix could be exactly there and solve another one of their problems at the same time: content-addressed outputs.

Warpforge -- project website: http://warpforge.io / https://github.com/warptools/warpforge -- is a project I work on that's heading a bit in this direction. Hashes in + build instruction = hashes out. It's really a powerful primitive indeed.

Building big sharable package systems is indeed a second half of the problem. Figuring out how to make the system have the hashy goodness AND be usefully collaborative with async swarms of people working independently and yet sharing work: tricky. We're trying to do it, though!

We have our first few packages now published, here: https://catalog.warpsys.org/

You can also see _how_ we build things, because we publish the full rebuild instructions with each release: for example, here's how we packaged our bash: https://catalog.warpsys.org/warpsys.org/bash/_replays/zM5K3V...

I'm in #warpforge on matrix with some collaborators if anyone's interested in joining us for a chat and some hacking :)

Gentoo supports building and installing binary packages from portage: You can compile once and then distribute the resulting package to other computers where they can be installed by Portage. Portage even verifies that the USE flags are the same!

It's called nix

NixOS uses a cache that works like this; it's source-based like Gentoo but for packages in cache it will download the cache rather than rebuilding. (it will rebuild if you configure the package in such a way to invalidate the cache, or install a package not in cache)

Besides the global cache for the whole distro, you can also set up caches for other software. For example, if you build your projects with Nix, you can have a cache for your projects (so that new contributors won't need to recompile everything from scratch). That's the premise behind https://www.cachix.org/

The only difference is that Nix caches aren't fine grained like ccache and sccache

Reply on news.ycombinator.com