Hey HN! We're launching a new cache storage backend for Depot today and wanted to share. Previously we stored container build cache on EBS volumes, where each customer project received its own EBS volume. That had some limitations, including fixed size, low performance, and subpar economics.

We're now using Ceph, running a storage cluster in top of i3en instances with local storage. This lets us thin-provision the cache storage volumes, elastically grow them as needed, and deliver much better IOPS and throughput compared to the gp3 volumes we used to use. Some very large builds that used to take as high as 8 minutes to construct image layers now take less than a minute!

For the most part, Ceph has been very seamless to deploy / operate, though we've been learning many fun facts about low-level Linux filesystems. :)