jacobwg

Hey, we are building our base image from the runner-images repo! I'll send you an email!

At the moment, I think you'd want to use both products together, i.e. using `depot build` in place of `docker build` to move the container build portion to a Depot container builder.

I'd like to have a more automatic integration at some point - the challenge is that a lot of BuildKit's architecture performs best when many different build requests all arrive at a single build host, it is then able to efficiently deduplicate and cache work across all those build requests. So you really want the many different Actions jobs all communicating with the same BuildKit host.

We have some ideas for reducing the amount of change to Actions workflows to adopt ^ - longer term we're also working on our own build engine, to free those workloads from being confined to single hosts (be that single CI runners or single container builders).

Hey, yep we have this today! Depot's original product is a fully-managed container build service (https://news.ycombinator.com/item?id=34898253) that caches all BuildKit layers to SSDs, so there's no cache-to/cache-from and cache doesn't need to transfer over the network at all.

Our original version of that system used vanilla BuildKit + EBS volumes + orchestration, nowadays we've replaced EBS with a distributed ceph storage cluster for significantly faster IOPS and throughput and have modified BuildKit to be better suited for high-performance distributed builds.

Both the container build service and the Actions runners are in the same AWS VPC, so they get good network performance between the two and don't need to egress over the internet.

We do get the promised throughput, but it depends on the destination as you've discovered. AWS actually has some docs on this[0]:

- For instances with >= 32 vCPUs, traffic to an internet gateway can use 50% of the throughput

- For instances with < 32 vCPUs, traffic to an internet gateway can use 5 Gbps

- Traffic inside the VPC can use the full throughput

So for us, that means traffic outbound to the public internet can use up to 5 Gbps, but for things like our distributed cache or pulling Docker images from our container builders, we can get the full 12.5 Gbps.

[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...

Thank you!

Yeah, GitHub's runners, especially the ones with >2 CPUs, have surprisingly long start times!

Yeah this is a good option if you'd like something to deploy yourself! You can also build an AMI from GitHub's upstream image definition (https://github.com/actions/runner-images/tree/main/images/ub...) if you'd like it to match what's available in GitHub-hosted Actions.

With Depot, we're moving towards deeper performance optimizations and observability than vanilla GitHub runners - we've integrated the runners with a cache storage cluster for instance, and we're working on deeper integration with the compute platform that we built for distributed container image builds - as well as expanding the types of builds we can process beyond Actions and Docker, for instance.

But different options will be better for different folks, and the `philips-labs` project is good at what it does.

We don't re-use the VMs - a VM's lifecycle is basically:

1. Launch, prepare basic software, shut down

2. A GitHub job request arrives at Depot

3. The job is assigned to the stopped VM, which is then started

4. The job runs on the VM completes

5. The VM is terminated

So the pool exists to speed up the EC2 instance launch time, but the VMs themselves are both single-tenant and single-use.

Yeah 5 seconds is from stopped to running, but to get that speed we need to pre-initialize the root EBS volumes so that they're not streaming their contents from S3 during boot. The GitHub Actions runner image is 50GB in size _just_ from preinstalled software!

We do support Graviton! I actually _just_ enabled them today, which we're calling "beta" for the moment: https://depot.dev/docs/github-actions/overview#depot-support....

The challenge with Arm is actually just that GitHub doesn't have a runner image defined for Arm. For the Intel runners, we build our image directly from GitHub's source[0], and we're doing the same for the Arm runners by patching those same Packer scripts for arm64. It also looks like some popular actions, like `actions/setup-*`, don't always have arm support either.

So the disclaimers for launching Depot `-arm` instances at the moment is basically just (1) we have no idea if our image is compatible with your workflows, and (2) those instances take a bit longer to start.

On achieving fast startup times, it's a challenge. :) The main slowdown that prevents a <5s kernel boot is actually EBS lazy-loading the AMI from S3 on launch.

To address that at the moment, we do keep a pool of instances that boot once, load their volume contents, then shutdown until they're needed for a job. That works, at the cost of extra complexity and extra money - we're experimenting some with more exotic solutions now though like netbooting the AMI. That'll be a nice blog post someday I think.

[0] https://github.com/actions/runner-images/tree/main/images/ub...

We're both offering managed GitHub Actions runners - some of the differences include:

- Depot runners are hosted in AWS us-east-1, which has implications for network speed, cache speed, access to internet services, etc. (BuildJet is hosted in Europe - maybe Hetzner?)

- Also thanks to AWS: each runner has a dedicated public IP address, so you're not sharing any third-party rate limits (e.g. Docker Hub) with other users

- We have an option to deploy the runners in your own AWS account or VPC-peer with your VPC

- We're integrating Actions runners with the acceleration tech we've built for container builds, starting with distributed caching

Yep, it's a good question! At the moment, my thoughts are roughly:

GitHub's incentives and design constraints are different than ours. GitHub needs to offer something that covers a very large user-base, to cover the widest possible number of workflows, and they've done this by offering basic ephemeral VMs on-demand. CI and builds are also not GitHub's primary focus as an org.

We're trying to be the absolute fastest place to build software, with a deep focus on achieving maximum performance and reducing build time as much as possible (even to 0 with caching). Software builds today are often wildly inefficient, and I personally believe there's an opportunity to do for build compute what has been done for application compute over the last 10 years.

GitHub Actions workflows are more of an "input" for us then (similar to how container image builds have been), with the goal of adding more input types over time and applying the same core tech to all of them.

You can! The default is that we launch the runners on our AWS account, but we do also have a bring-your-own-cloud deployment option.

We have some docs on this for our container builder product - still need to write the docs for Actions runners too, though they use the same underlying system: https://depot.dev/docs/self-hosted/overview.

Wireguard's lower-level than the solutions that were benchmarked - in fact, Netmaker and Tailscale themselves use Wireguard as their backing technology.

But even with them both using Wireguard, there are choices involved that affect performance, for instance whether to use the Wireguard kernel model or userspace implementation, how to configure routing, packet filtering, firewalls, etc.

Not sure how common the use-case is, but we're using Ceph to effectively roll our own EBS inside AWS on top of i3en EC2 instances. For us it's about 30% cheaper than the base EBS cost, but provides access to 10x the IOPS of base gp3 volumes.

The downside is durability and operations - we have to keep Ceph alive and are responsible for making sure the data is persistent. That said, we're storing cache from container builds, so in the worst-case where we lose the storage cluster, we can run builds without cache while we restore.

ClickHouse is a popular open-source analytics database (https://github.com/ClickHouse/ClickHouse), so in this case they are assuming you know what ClickHouse is yes.

I've seen a hybrid where you have a form with a button to confirm, but include JavaScript to auto-submit the form on load. For the crowd that has JS disabled, they can click the button, but otherwise it's one-click from the email.

No idea if this holds if/when the email crawler bots start executing JS on crawl.

Hey HN, I wrote this about how we've been reducing the time to start running container builds at Depot (YC W23) largely through optimizing EC2 instance start time - primarily we found that stopped EC2 instances start a lot faster than new ones.

Happy to answer any questions / provide any additional details!

We encountered a specific GitHub Actions restriction at Depot[0]: for pull_request workflows that originate from open-source forks, Actions disables access to all repository secrets and to the Actions OIDC issuer, as a security mechanism to deny untrusted code access to those secrets.

But we needed a way to authenticate our CLI within those public workflows. This OIDC issuer is the result of that need, and works like so:

1. The pull_request workflow makes a "claim request" to the OIDC issuer, claiming certain details about the workflow like the ID, run ID, repository, etc.

2. The OIDC issuer responds with a "challenge code" that the workflow must periodically print to its logs

3. The OIDC issuer connects to the GitHub Actions websocket endpoint for log streaming, validates that the challenge code is being printed, then returns a new OIDC token to the workflow

This is working well for us, and lets us acquire an OIDC token similar to the GitHub Actions native OIDC token. The issuer itself runs as a Cloudflare Worker.

Happy to answer questions and I'd love any feedback you may have!

[0] https://depot.dev

Hey HN! We're launching a new cache storage backend for Depot today and wanted to share. Previously we stored container build cache on EBS volumes, where each customer project received its own EBS volume. That had some limitations, including fixed size, low performance, and subpar economics.

We're now using Ceph, running a storage cluster in top of i3en instances with local storage. This lets us thin-provision the cache storage volumes, elastically grow them as needed, and deliver much better IOPS and throughput compared to the gp3 volumes we used to use. Some very large builds that used to take as high as 8 minutes to construct image layers now take less than a minute!

For the most part, Ceph has been very seamless to deploy / operate, though we've been learning many fun facts about low-level Linux filesystems. :)

Do you have a link or serial number for the application?

Do you have a trademark on the word "throw"? The page title is currently "Throw™" and many of the instances of Throw have the (tm), but I can only find the trademark application for your logo design (serial 97072015).

I mean, I took Sindre's move to ESM-only publishing as his own choice as the author and maintainer of his own OSS packages. In the spirit of OSS, he doesn't need to justify the choice, it's good enough that as the author he's making it. The previous CJS versions are still published, and anyone can fork the modules if they want to take on the maintenance burden of the CJS version.

That preference has had an impact on the Node ecosystem at large, given how prolific an OSS contributor Sindre is, but IMO that influence has been earned by the large body of work he's contributed.

I think Kyle answered this below, about the option for enterprises to run the data plane of Depot in their own cloud account. In that model, the Depot CLI connects directly to that data plane without passing through any infrastructure on our side.

> I think we solved it with buildkit cache

One big thing we're doing here, if you're familiar with BuildKit cache, is providing builds a stable cache SSD that's reused between builds. This means we support all of BuildKit's caching features, including things like cache mounts that aren't directly supported in ephemeral CI environments. Plus Depot doesn't need to save or load the cache to a remote store like S3 or GitHub Actions cache, instead the previous cache is immediately available on build start.

This may not be any better or different than what you're doing, I just wanted to mention the detail for anyone familiar with trying to make BuiltKit more performant.

In AWS - we launch either Intel or Arm EC2 instances depending on the requested build platform (or both for multi-platform builds). When a project's builds are running, they have sole control of that instance, which is terminated when the builds are done.

To make this performant we keep a certain number of spare "warm" machines ready for build requests so that you don't have to pay the instance launch time penalty yourself.

It should yeah, our builders are based on BuildKit rather than Kaniko, which optimizes for building container images in parallel and caching as much as possible. BuildKit also supports some more advanced types of caches, such as cache mounts: https://github.com/moby/buildkit/blob/master/frontend/docker...

Both Kaniko and BuildKit can be run in rootless mode - we are not doing this, instead we give every builder access to an isolated VM, so builds are a bit quicker as well by avoiding some of the security tricks that rootless needs to work.

Travis CI got acquired by Idera in 2019 (https://news.ycombinator.com/item?id=18978251) then a month later laid off senior engineer staff (https://news.ycombinator.com/item?id=19218036).

Nice, yeah let me know if you want to chat further, feel free to send an email to jacob@depot.dev or the address in my profile. It would be really useful for us to know more about how you have things structured in GCP, etc.

The docs can be improved, but here's the GitHub Actions guide: https://depot.dev/docs/integrations/github-actions

The most important bit though is that we have a `depot/build-push-action` that implements the same inputs as Docker's `docker/build-push-action`, so just swapping that line and adding a project ID and access token are the majority of what you'd need to do:

      - uses: depot/setup-action@v1
      - uses: depot/build-push-action@v1
        with:
          project: <your-depot-project-id>
          token: ${{ secrets.DEPOT_TOKEN }}

          # Whatever other inputs:
          context: .
          push: true
          tags: |
            ...

I think that's along the lines of what you're describing as a Depot GitHub Action: https://github.com/depot/build-push-action.

We actually briefly entertained the idea of adding GitHub Runners as well. :) Perhaps someday, but we have a long way to go just on just Docker / containers.

Just to note, you can totally use Depot within your GitHub Actions runs, even if those runs are happening inside self-hosted or BuildJet-hosted runners. You might get the best of both worlds that way, having your builds and tests outside of Docker run on BuildJet, and Docker builds accelerated with Depot.