Hacker News Reader

Comment by jacobwg | Hacker News Reader

Yes, all the layer cache is saved on a persistent volume that's reused between jobs. In that respect, it's very similar to running `docker build` on your local laptop, where each build is incremental. But in Depot's case, that incremental experience is shared for all CI builds, and between all users running `depot build` on their local devices.

View on HN

It sounds like the SSD provides the layer cache with `--cache-to=type=local`? Does Depot support connecting additional VMs to the volume if someone wants to scale up? I've been meaning to look into the new S3 cache to solve this issue.

Exactly, we effectively persist BuildKit's state directory, so it doesn't even need to tar and export any of the internal state or layer cache, it gets the same state volume for the next build.

Today we're using vertical scaling, so we run BuildKit on larger EC2 instances, tune `max-parallelism`, and let BuildKit handle processing multiple builds / deduplicating steps across builds / etc. What our website calls a "project" basically equates to a new EC2 instance (so two different projects are fully isolated from each other).

We'd like to expand into horizontal scaling, probably with some kind of tiered caching, so the builders would be able to use local cache on local SSDs if available, but fall back to copying cache from S3 if not available locally.

Awesome. How does this work in a distributed sense? For example, for multiple parallel builds where each build is building a different git sha?

Today we're vertically scaling builders, so multiple builds run on the same large EC2 instance. The instances support processing multiple Docker builds at once, and thanks to BuildKit can even deduplicate work between multiple parallel builds, so like if two simultaneous builds share some steps, it can handle coordinating so those steps run just once.

We have plans to expand to more horizontal scaling with tiered caching, so we can keep the speedups we see today but further increase potential parallelism.

Yes! Love the vertical scaling approach. Pinning similar builds to same host doesn't get enough love.

Reply on news.ycombinator.com