Mozilla sccache: ccache with cloud storage
github.com
github.com
[build]
rustc-wrapper = "/path/to/sccache"
We just released S3 and Redis support. https://less.build
Buildless also supports Gradle, Maven, Bazel, CCache and Turbo
sam@elide.dev is fine too, thank you deeply for reporting this to us
Seriously, thank you!
Re/spoofing, obviously it's all protected with API keys and tokens, and we're working on mechanisms to perform end-to-end encryption. In general, build cache objects are usually addressed by a content-addressable-hash, so that also helps because your build typically knows the content it's looking for and can verify.
That isn't true for all tools, though, so we're working to understand where the gaps are and fix them.
How does that work? I would think the simplest case of a build object that needs to be cached is a .o file created from a .c file. The compiler sees the .c file and can determine its hash, but how can the compiler determine the hash of the .o file to know what to look up in the cache? I think the compiler would need to perform the lookup using the hash of the .c file, which isn't a hash of the data in the cache.
Of course one other advantage to build caches is they are verifiable: the intent is to produce the exact same output as a normal call, and that’s easily checked on the client side.
No question that build caching poses inherent supply chain risks though and that’s part of what we want to solve. I think people are hesitant to trust build caching for good reason until there are safer mechanisms and better cryptographic patterns applied.
Most importantly you can look up an action in the ActionCache without actually running it, provided you have the inputs at hand. So now when another person comes by and runs the same build command, they say "Has this Action, with these inputs, been run before?" and the server can say "Yes, and the output is a file identified by hash XYZ" where XYZ is the hash of foo.o, so you can just instantly download it from the CAS.
So there are a few more moving parts to make it all work. But the system really is ultimately content-addressed, for the most part.
[1] https://github.com/bazelbuild/remote-apis/blob/main/build/ba...
If you don't want clients to have to trust each other, then you can block ActionCache write access to the clients and add remote execution. In this setup clients upload an action to the CAS, remote executors run the action and then upload the result to the ActionCache, using the hash of the action as the key. This way malicious clients can't spoof cache results for other clients, because other clients won't ever look for the malicious action's key in the ActionCache.
Explained better than I could for sure. I find it very interesting how BuildXL and Bazel ended up at similar models for this problem. I don’t yet know the history of which informed which.
(As compared to, say, Gradle, which works based on input hashes instead.)
By spoofing I meant to say that an authenticated but malicious client (intentionally or not, e.g. a clueless intern) may be able to write malicious contents to the cache. For example, their build toolchain could be contaminated and the resulting build outputs are contaminated. The "action" per se and its hash is still legit, but the hash is only used as the lookup key -- their corresponding value is "spoofed."
The only safe way I can imagine to use such a remote cache is for CI to publish its build results so that they could be reused by developers. The direction from developers to developers or even to CI seems difficult to handle and has less value. But I might be missing some important insights here so my conclusion could be wrong.
But if that's the case, is the most valuable use case to just configure the CI to read from / write to the remote cache, and developers to only read from the remote cache? And given such an assumption, is it much easier to design/implememt a remote cache product?
One goal of these tools is to guarantee that such misconfiguration results in a cache key mismatch, rather than a hit and a bug.
There are tons of challenges designing a remote build cache product, like anything, but that one has turned out to be a reliable truth.
Some other interesting insights:
- transmitting large objects is often not profitable, so we found that setting reasonable caps on what’s shared with the cache can be really effective for keeping transmissions small and hits fast
- deferring uploads is important because you can’t penalize individual devs for contributing to the cache, and not everybody has a fast upload link. making this part smooth is important so that everyone can benefit from every compile.
- build caching is ancient, Make does its own simple form of build caching, but the protocols for it vary in robustness greatly, from WebDAV in ccache to Bazel’s gRPC interface
- most GitHub Actions builds occur in a small physical area, so accelerating build artifacts is an easier problem than, say, full blown CDN serving
The assumptions that definitely help:
- it’s a cache, not a database; things can be missing, it doesn’t need strong consistency
- replication lag is okay because a build cache entry is typically not requested multiple times in a short window of time; the client that created it has it locally
- it’s much better to give a fast miss than a slow hit, since the compiler is quite fast
- it’s much better to give a fast miss than an error. You can NEVER break a build; at worst it should just not be accelerated.
It’s an interesting problem to work on for sure.
Nix works this way by default (and much of the community operates caches like this) and it can be a massive, massive time saver.
> How do you make sure the build results are not spoofed?
What do you mean "spoofed?" As in, someone put an evil artifact in the cache? Or overwrote an existing artifact with a new one? Or someone just stole your developers access and started shoving shit in there? There's a whole bunch of small details here that really matter to understand what security/integrity properties you want the cache to uphold.
FWIW, I've been looking into this in Buck2/Bazel land, and my understanding is that most large orgs just use some kind of terminating auth proxy that the underlying connection/flow/build artifacts can be correlated back to. So you know this cache artifact was first inserted by build B, done by user X, who authenticated with their key K, etc etc.
Nix + sccache would probably be pretty great for preserving paths and environment, which is really healthy for build caching in general.
Cargo is a good example. For a number of (practical) reasons, cargo -> nix translators are a lot of effort and often have bugs, so for "simplicity" all upstream crates just compile every crate dependency every time. That means if two crates use the same dependency, it gets compiled twice. It's important to understand this is no worse than the way Cargo works already for most people. Cargo does not have a content address storage model, unfortunately. But it's pretty annoying for Nix users and costs a lot.
In theory, we could wrap rustc with a recursive-nix enabled wrapper so that rlib's etc each get a granular derivation and then get put into the host store. So assuming a crate gets built with the same flags, between two "outer" expressions, they'll get to share the work and it will go into the store. A working example of this for C++ code is nix-ccache, but a fully robust implementation is a bit of work.[1]
Recursive Nix is still experimental but there is some use of it (privately and publicly) that I know of for these purposes.
I just tried it for the first time the other day and although I’m not ready to move to it yet, I can already see the brilliance.
I wasn't really referring to the coarseness though; just that a lot of Nix projects provide build caches to speed things up for contributors. It's not just something for internal teams. And it really does help.
I've set it up at work with two S3 buckets: trusted and untrusted. CI/CD read/write from trusted only. Developers read/write from untrusted, and read-only from trusted.
Or, maybe the blobs you’re dealing with are on the bigger end? That would also make sense
It's certainly much faster to download the .o than it is to build it. Once it's downloaded it stays on the local filesystem until it's garbage-collected.
R2 is S3 to me, btw, or at least it’s the same API. Object storage as a model is really what I’m asking about. I’m genuinely glad to hear S3 was enough to be impactful.
But it looks like it could be competitive pricing-wise as well, especially if you need to access the same data across regions.
That depends on your network latency and throughput, plus the performance of your compiler.
With fast compilers and less bloat (not C++ nor Rust) such a distcc is certainly slower.
We would see taking over execution of your build as much more dangerous.
No question though that build caching in shared form, in SaaS form, needs extra special attention paid to security. Our product doesn’t introspect cache blobs and in fact doesn’t really want to. Once we figure out how to make the crypto work, we shouldn’t be able to see any of that data at all.
Access can be made public for reads (OSS) but is always identified for writes.
I’ve found that remote execution ends up returning much less benefit than remote caching, but that’s just me and it’s entirely possible I Did It Wrong the whole time)
The intent with Buildless is to release a free-first toolchain that helps with build caching in earnest and makes the whole problem much less error prone. Then the Cloud stuff on top is for groups who need more gas. Cloudflare is generously supporting our upcoming free tier.
Could you expand a bit please? As somebody who is too cheap to get more than the 256gb MacBook, those heavy ~1gb random target/ directories add up over time. Didn't know there was an easy/cheap way around this?
export CARGO_TARGET_DIR=/absolute/path/to/target
Though beware that projects that share a target dir can't build simultaneously. If you have multiple projects open that can get annoying, because it can also effect the language server. I'm that case you can use gentler measures like cargo-sweep.
(And if you're using Bazel or one of its brethren then they generally have native remote caching and execution support.)
Meanwhile, the only legitimate problem you mentioned is if it causes a build error and we don't immediately consider it as the source of the issue. But I use the check command so often that it is easy to suspect the cache if check succeeds and build fails.
You may be lucky and be building a pure or mostly pure Rust program, in which case it works pretty well. Throw in some C/C++ and it starts to degrade (though it's still better than with an actual C/C++ program because you aren't actually editing the C/C++ code generally).
And are you actually using remote caching/compilation? Because there's no way you can set that up in 5 minutes.
Who said anything about remote caching? You don't need it to benefit from it. It's useful if you build a lot of Rust code. A lot of packages turn up repeatedly as dependencies among several projects.
> You may be lucky and be building a pure or mostly pure Rust program, in which case it works pretty well. Throw in some C/C++ and it starts to degrade
You're making assumptions again. What is the issue with pure Rust code? Rust isn't like Python needing C or C++ support for process-intensive parts. And much of the C/C++ dependencies are dynamically-linked, with Rust wrappers. I haven't seen many projects that require Rust and C/C++ code to be built together and statically linked.
Besides, I haven't heard anyone complain about sccache that much. How prevalent is the degradation anyway?
Knowing the internals of some of them, I’ve found that build cache clients are way more likely to miss with a cache key misalignment than they are to mixup two objects. I’m sure it’s possible, I’ve just never seen it happen in the wild, after extensive usage.
Generally speaking these tools are very conservative about two inputs matching: an identical file at a different path will cause a cache key change in sccache.
You're underselling it, honestly. For those who haven't looked into it, if you want to enable it for all Rust projects on your system, literally all you need is install the binary and then add this to ~/.cargo/config, and then it will be enabled whenever you invoke `cargo` (or even rustc directly iirc) from the user who's config file you modified)
[build]
rustc-wrapper = "/path/to/sccache"
Is sccache susceptible to cache misses when using full paths as cache keys? It would be very helpful if compiling “/home/heads/project/foo.c” could use the cached result of compiling “/home/thunderbong/project/foo.c”.
Then in 2016 it seems like sccache was re-implemented in Rust (https://github.com/mozilla/sccache/commit/3da89195ce91a576cc...), from the initial Python implementation.
NFS isn't horrible, though it's limited. OTOH object storage is limited but has some other advantages. So there needs to either be the latter as a standard that can be adopted by all vendors, apps and OSes, or a new standard that fills the gaps of what apps want.
RCE to a new level. Trust us, it is secure.
When any unique piece of Unison is compiled by anyone, it no longer needs to be compiled by everyone.
As a result, we impose reasonable caps which force a recompilation for the largest outputs, and it ends up being faster.
Just our experience and of course mileage varies by project. Redis can do 512mb values but the cap needed for efficiency’s sake is much smaller than that.