>...persistent disks significantly lowers build time
Does this mean your solution places specific caches, like bazel, node_modules, .yarn, and other intermediary artifacts onto a shared volume and reuses them among jobs?
Today we're using vertical scaling, so we run BuildKit on larger EC2 instances, tune `max-parallelism`, and let BuildKit handle processing multiple builds / deduplicating steps across builds / etc. What our website calls a "project" basically equates to a new EC2 instance (so two different projects are fully isolated from each other).
We'd like to expand into horizontal scaling, probably with some kind of tiered caching, so the builders would be able to use local cache on local SSDs if available, but fall back to copying cache from S3 if not available locally.
We have plans to expand to more horizontal scaling with tiered caching, so we can keep the speedups we see today but further increase potential parallelism.
Now as an end user you still have to add this to your Dockerfile, but if subsequent builds are able to continually use this cache, build times will drastically improve.
One of the reasons we created Depot is that cache mounts aren't supported in GitHub Actions, since each CI run is entirely ephemeral, so the files saved in a cache mount aren't saved across runs. BuildKit doesn't export those cache types via cache-to. There are some manual workarounds creating tarballs of the BuildKit context directory, but we wanted something that just works without needing to save/load tarballs, which can be quite slow.
1. If you need an uncached individual build, you can pass the `--no-cache` flag to build everything from scratch
2. We have a "clear all cache" button in our UI and CLI command, to wipe the entire project's cache disk.