Hey, yep we have this today! Depot's original product is a fully-managed container build service (https://news.ycombinator.com/item?id=34898253) that caches all BuildKit layers to SSDs, so there's no cache-to/cache-from and cache doesn't need to transfer over the network at all.
Our original version of that system used vanilla BuildKit + EBS volumes + orchestration, nowadays we've replaced EBS with a distributed ceph storage cluster for significantly faster IOPS and throughput and have modified BuildKit to be better suited for high-performance distributed builds.
Both the container build service and the Actions runners are in the same AWS VPC, so they get good network performance between the two and don't need to egress over the internet.