We solve it 2 ways in the Bazel ecosystem: for the intermediate artifacts, we only fetch the digest (hash + size) of the blobs to calculate the merkle tree forward. The blob itself can stay on the remote cache server.

For the bigger final artifacts, we support using Content Defined Chunking (rolling gear hashing) to only fetch the missing chunks between incremental builds. Binaries executable with stable layout benefits from this quite a lot.

We are definitely not done with all of the improvements here. But since all the major AI labs are using Bazel, we know that the tools can support “Agent Scale”. https://webazel.dev/