jacobwg

We have two ways at the moment:

1. If you need an uncached individual build, you can pass the `--no-cache` flag to build everything from scratch

2. We have a "clear all cache" button in our UI and CLI command, to wipe the entire project's cache disk.

This is good advice!

> Use Docker layer caching on GitHub Actions or your favorite CI (https://evilmartians.com/chronicles/build-images-on-github-a...)

Since the time that article was written, BuildKit added native support for interacting with the GitHub Actions cache, so it's even easier than they describe:

    - uses: docker/build-push-action@v4
      with:
        cache-from: type=gha
        cache-to: type=gha,mode=max

Kyle actually wrote a blog post with more details if you're interested: https://depot.dev/blog/docker-layer-caching-in-github-action...

What has to happen with cache-to and cache-from is BuildKit has to create cache tarball files for each of the layers to be cached, then upload them to the remote cache destination. Then next build when there's a cache match, BuildKit downloads the tarballs from the remote store and un-tars them so that it can continue the build.

This process can be slow, from creating the tarballs, to transferring them over the network, to unpacking them again. This is one of the reasons we created Depot in the first place. We had several commercial projects with highly optimized Dockerfiles, where saving and loading cache to GitHub Actions took several minutes, representing a significant amount of the overall build time. We also wanted to use some more advanced Dockerfile features like `RUN --mount=type=cache` that are entirely unsupported in GitHub Actions.

With Depot, builds re-use the same persistent SSD, so you get perfectly incremental performance, similar to the speed you'd get on a local machine, without any network transfers. So if you put effort into optimizing Dockerfiles as you describe, you actually get to keep all that time savings. Our app backend's CI Docker builds usually complete in about 20 seconds for instance.

> hadolint is a good docker linter but there is no tool in the market that helps people optimize docker images

We're just getting started on this, but we'd like to provide insights like this through Depot, since we control the build execution environment and can observe what happens inside the build.

Today we're vertically scaling builders, so multiple builds run on the same large EC2 instance. The instances support processing multiple Docker builds at once, and thanks to BuildKit can even deduplicate work between multiple parallel builds, so like if two simultaneous builds share some steps, it can handle coordinating so those steps run just once.

We have plans to expand to more horizontal scaling with tiered caching, so we can keep the speedups we see today but further increase potential parallelism.

Super cool, if you'd ever like to chat, let me know, my email is in my profile. We have an API in private beta to dynamically manage namespaces and acquire BuildKit connections within those namespaces, designed for platforms that need to build images on behalf of their customers. Any feedback you might have would be awesome!

I haven't used Azure DevOps scale sets, but it looks similar, assuming it is orchestrating a persistent cache disk.

If you're not using Azure DevOps, or even if you are, you can use `depot build` from any CI provider or from your local machine, anywhere you'd run `docker build` today, and the speed and cache are shared everywhere. And you don't need to configure any infrastructure. We also orchestrate both Intel and Arm machines depending on what type of build you need to do (or both for multi-platform). That can be important if you need to run on Arm in production, or if you have developers using new MacBooks.

We're working on more Docker-specific features as well, we plan to directly integrate things like SBOMs, container signing, build analytics and insights for what happens inside the Docker build, etc into Depot, there are some interesting things we can do having control of the execution environment.

We'd like to expand to support GCP as well yes. Are you looking to run the Depot runners in your own GCP account?

So we have the concept of a "project", which in retrospect isn't the best name and is way too vague. :) But on our end, a "project" equates to one or two cache SSDs + one or two EC2 instances we're running, depending on whether you've asked for a single platform build or an Intel+Arm multi-platform build.

We do charge $0.05 per minute of build time used, but in theory that $49/mo plan gives you access to up to 20 build machines, if you're building 10 projects at once.

That said, if you already have your own dedicated build cluster / CI setup, you may prefer to just use that! Depot is effectively doing that kind of thing for you if you don't already have your own hosted CI system or would prefer not to orchestrate Docker layer cache.

We will be expanding to more things like SBOMs, container signing, insights and analytics about what's happening inside the builds, but hopefully in more integrated ways, since we control the execution environment itself.

Potentially - if Nix or Bazel has already built the binaries, and just needs to construct an OCI-compliant image tarball with them, that can be quite quick, similar to a Dockerfile with only COPY instructions. Nix and Bazel can also give you deterministic / reproducible images that take more effort to construct with Dockerfiles.

I've also seen people use Nix or Bazel inside their Dockerfile, like ultimately the build has to execute somewhere, be that inside or outside a Dockerfile.

Exactly, we effectively persist BuildKit's state directory, so it doesn't even need to tar and export any of the internal state or layer cache, it gets the same state volume for the next build.

Today we're using vertical scaling, so we run BuildKit on larger EC2 instances, tune `max-parallelism`, and let BuildKit handle processing multiple builds / deduplicating steps across builds / etc. What our website calls a "project" basically equates to a new EC2 instance (so two different projects are fully isolated from each other).

We'd like to expand into horizontal scaling, probably with some kind of tiered caching, so the builders would be able to use local cache on local SSDs if available, but fall back to copying cache from S3 if not available locally.

--mount=type=cache is awesome!

One of the reasons we created Depot is that cache mounts aren't supported in GitHub Actions, since each CI run is entirely ephemeral, so the files saved in a cache mount aren't saved across runs. BuildKit doesn't export those cache types via cache-to. There are some manual workarounds creating tarballs of the BuildKit context directory, but we wanted something that just works without needing to save/load tarballs, which can be quite slow.

Yes, all the layer cache is saved on a persistent volume that's reused between jobs. In that respect, it's very similar to running `docker build` on your local laptop, where each build is incremental. But in Depot's case, that incremental experience is shared for all CI builds, and between all users running `depot build` on their local devices.

Hey, I'll look into this one - this kind of error usually means that the BuildKit server believes the build has finished, but the CLI missed the fact that the build ended. It _might_ be related to your Rust build, but I want to make sure there's not something happening on our end.

Hey, we support that exact thing, the `depot` CLI wraps buildx so we support the same `--platform` flag.

The difference between Depot and doing it directly in GitHub Actions is the "native CPU" part - on GitHub Actions, the Arm builds will run with emulation using QEMU. It works, but it's often at least 10x slower than it could be if the build was running on native Arm CPUs. Builds that should take a few minutes can take hour(s).

For multi-platform, if you want to keep things fast, you need a simultaneous connection to both an Intel and an Arm machine, and the two work in concert to build the multi-platform image.

There are workarounds with emulation, or pushing individual images separately and merging the result after they're in a registry. But if you just want to `build --platform linux/amd64,linux64` and it be fast, we handle all the orchestration for that.

Hey! This is cryptominer protection - we used to have an option to access a trial without a credit card, and the cryptominers did indeed sign up. :) Requiring a credit card for the free tier is a speed bump for that, on the free plan it subscribes you to a $0 product in Stripe.

> What happens when GitHub adds native arm support though?

That will make it much faster to build arm images on GHA natively - in that scenario, Depot should still be several times faster like we are on Intel today, primarily due to how we're managing layer cache to avoid needing to save it or load it between builds (cache-to / cache-from), as well as just having larger runners and more sophisticated orchestration. We can take advantage of BuildKit's ability to share cache and deduplicate steps between concurrent builds for instance.

We're also expanding Depot in a few different directions, including along the security path with container signing and SBOM support, as well as some upcoming build insights and analytics features. The goal is that it's always super easy to `depot build` from wherever you `docker build` today, and that Depot provides the best possible experience for container builds.

> could somebody avoid the traditional `docker build` with say NixOS or Bazel

Yes! You can think of an OCI image as a special kind of tarball, so things like NixOS and Bazel are able to construct that same tarball, potentially fairly quick if it just has to copy prebuilt artifacts from the store.

Today we're running BuildKit, so we support all the typical Docker things as well as other systems that use BuildKit, e.g. Dagger, and I believe there are nix frontends for BuildKit. In that sense, we can be an accelerated compute provider for anything compatible with BuildKit.

> build Docker images faster than Docker

Today the trick is in the hosting and orchestration. We're using fast machines, launching Graviton instances for Arm builds (no emulation) or multiple machines for multi-platform build requests, orchestrating persistent volumes, etc. It's more advanced than what hosted CI providers give you today, and closer to something you'd need to script together yourself with your own runners. There's also some Docker build features (e.g. cache mounts) that _only_ work with a persistent disk.

For me, the happy medium is an extension that replaces thumbnails with a frame from the actual video itself (https://chrome.google.com/webstore/detail/clickbait-remover-...).

Videos are still visually identifiable this way, and as a plus you get a preview of their actual contents, all while significantly cutting down on visual noise.

The version installable with Homebrew is working for me on Ventura / M1.

I have an occasional issue where one of my external displays switches from 60 Hz to 40 Hz, and I use displayplacer to toggle the refresh rate to reset the monitor back to 60: https://github.com/jacobwgillespie/dotfiles/blob/main/bin/fi...

Anecdotally, this is starting to shift with M1 MacBooks, Graviton is looking more attractive for precisely that architecture parity reason for teams using majority M1 devices.

Agreed, this is more or less the inspiration behind Depot (https://depot.dev). Today it builds Docker images with this philosophy, but we'll be expanding to other more general inputs as well. Builds get routed to runner instances pre-configured to build as fast as possible, with local SSD cache and pre-installed toolchains, but without needing to set up any of that orchestration yourself.

I would love to see more development in this space. I'm aware of tools like bazel, nix, and jib (I feel like I've seen an equivalent to jib for Go) that can build Docker images "outside" Docker/BuildKit. I believe the .NET SDK can do something similar for .NET applications. And then there's tools like buildpacks, nixpacks, devbox, etc that still use Docker to produce the image, but don't require thinking in Dockerfiles.

I have some ideas I want to explore with Depot, but it really it feels like it should be possible to make cross-compiling and dependency encapsulation easy to use, so "building a container image" is reduced to creating a tarball in the correct OCI format, and you don't have to interact with Dockerfiles as much if you'd prefer not.

It is! At least in terms of raw minute pricing, though Depot builds are often faster than Actions, around 2-5x for Intel builds and closer to 10x faster for Arm builds, which affects the comparison a bit.

Our biggest cost at the moment is persistent disks. Each project gets its own SSD(s) that live for the lifetime of the project, and that's a major part of why Depot builds are so much faster and support BuildKit features that require persistent disks (e.g. RUN --mount=type=cache). We have some ideas for how we might improve this in the future, keeping the same or better performance profile but avoiding the continuously-provisioned disk, we'll see how that goes.

One note on "per project", each Depot project represents a pair of Intel/Arm VMs, each with a 50GB SSD. But there's otherwise no restriction on what you build using those projects, you can use a single project with multiple repos/Dockerfiles/whatever. Projects are the access control boundary, and building multiple disparate images using the same project would cycle the cache faster, but otherwise there are no limits on concurrency or what's building inside those VMs.

Not currently - Depot runs BuildKit (https://github.com/moby/buildkit) on our builders, and it does not support Windows containers to my knowledge.

Possibly? I'm less familiar with building iOS applications, but Depot is providing a native Linux amd64 builder VM, so if you're able to build an iOS app in that environment, it should just work.

Hey, so a Depot "project" represents the two VMs / SSDs for building each architecture + access control in the form of project-scoped API tokens. But there's nothing preventing you from building all your small services using just a single project.

You very well could build everything on just the single $25 project, there aren't any restrictions on concurrency or what images you build on what projects.

It should be just as as straightforward with Depot as well - we have a depot/build-push-action (https://github.com/depot/build-push-action) that accepts the same inputs as docker/build-push-action to make it easy (under the hood, both use BuildKit).

    steps:
      - uses: depot/setup-action@v1
      - uses: depot/build-push-action@v1
        with:
          platforms: linux/amd64,linux/arm64

Nice, yeah we had one customer using Cloud Build, and in their case their native amd64 builds dropped from around 12 minutes to 1-2 minutes on Depot builders. Surprisingly to us, it was even faster to push from Depot to Artifact Registry than from Cloud Build directly, definitely an odd networking quirk.

arm64 builds should even be a more dramatic change, they should drop to at least as fast as your current amd64 builds.

Yeah Nixpacks is super cool! It uses nix and BuildKit to produce images for various different languages, and since it uses BuildKit under the hood, it might be able to use Depot's BuildKit to perform the actual build. I'd love to see what that integration would look like, Depot would provide the compute and persistent caching, and nixpacks would simplify the build process itself.

It's basically the same, just without you having to do any of the work to set up or maintain the remote machines.

When you start a container build with Depot, it will launch an Intel or an Arm VM running BuildKit (or both for a multi-platform image). It creates persistent SSD cache volumes for each project and attaches those to the build machines at launch. And it automates connecting your CLI to those machines over TLS. Then when the build is done, it shuts down those machines to save cost.

Depot does all that either in our cloud account, or you also have the option of connecting your own AWS account and Depot will act just as the orchestration layer.

So basically you get the remote build machines, but without needing to set them up, manage costs, configure access control, etc. But at its core, it's running the open-source version of BuildKit (the Depot CLI embeds the same Go library as `docker buildx build`).

I think we are trying to take on GitHub Actions' compute side yeah — you'd want to either self-host your own BuildKit or use something like Depot for two major reasons:

1. Several of BuildKit's caching features either require or work best with a local disk. For instance things like `RUN --mount=type=cache` need a persistent disk, which Actions doesn't provide. You can also hit bottlenecks with saving and loading tarball cache from Actions (e.g. cache-from / cache-to) where it takes longer to download and unpack the tarballs compared to omitting the cache. Having a persistent disk makes all this "instant".

2. If you need to build images for ARM CPUs, QEMU in GitHub Actions is really slow. And if you want to build multi-platform images natively (e.g. --platform linux/amd64,linux/arm64), BuildKit needs a simultaneous connection to two native builders. Depot automates all that (it will launch two VMs at once and connect them to the build).

GitHub Actions today also only offers 2 CPUs for the build - they're launching a beta of larger machine sizes currently.