A faster path to container images in Bazel | Hacker News Reader

A faster path to container images in Bazel

tweag.io

104 pointsmalt358 comments

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top.

I'm not too surprised that out of the box docker images exhibit more of this. While it's good they're fixing it, it feels like maybe some of the core concepts cause pretty systematic issues anytime you try to do anything beyond the basic feature set...

To be clear, when you say “they’re fixing this”…the Bazel maintainers have nothing to do with this.

Bazel is a general purpose tool like Make. But with caching and sandboxing and different syntax.

Make is no less focused on Docker than Bazel is.

Unlike Make however, Bazel does make it easy to share rule sets.

But you don’t need to use other people’s Bazel rule sets any more than you need to use other people’s Make recipes.

This author has a clever way to minimize needing to touch layers at all.

rules_oci (and bunch of rules_* under bazelbuild / bazel-contrib org on GitHub) is Bazel recommeded rule sets.

I don't agree with your parent comment about Bazel, but your comment is not fair too. Bazel tries to be better build tool so it took on responsibility on registry / rules_* and get critics for it is a fair game.

The "bloated Bazel" blame is not fair too, but I think somewhat understandable. If you ever going to only do JavaScript, bun or other package manager is enough and "lighter-weight". Same goes to uv + Python bundle. Bazel only shines if you are dealing with your C++ mess and even there, people prefer CMake for reasons beyond me.

For some more depth into the "bloat" of bazel, I like this reference: https://fzakaria.com/2024/02/27/hermetic-but-at-what-cost

Curious if this bloat is present in a Build Without Bytes scenario?

What can used instead for a large multilanguage repo where we want to build every commit?

Genuine question - also find Bazel frustrating at times.

Personally, I write my own build systems.

Any readily available build system is more of a meta-language onto which you code your own logic, but with limited control and capabilities. Might as well take control of the whole stack in a real programming language.

Building my own build system lets me optimize my workflow end-to-end, from modular version management, packaging and releasing, building and testing, tightly integrating whatever tool or reporting I want, all seamlessly under the same umbrella.

I mostly do C++, Assembly, eBPF, Python (including C++ Python modules), and multi-stage codegen on Linux, so I haven't really looked at the complexity of other languages or platforms.

I tried Basel, Buck2 and Pants for a greenfield mono repo recently, Rust and Python heavy.

Of the three, I went with Buck2. Maybe just circumstance with Rust support being good and not built to replace Cargo?

Bazel was a huge pain - broke all standard tooling by taking over Cargos job, then unable to actually build most packages without massive multi-day patching efforts.

Pants seemed premature - front page examples from the docs didn’t work, apparently due to breaking changes in minor versions, and the Rust support is very very early days.

Buck2 worked out of the box exactly as claimed, leaves Cargo intact so all the tooling works.. I’m hopeful.

Previously I’ve used Make for polyglot monorepos.. but it requires an enormous amount of discipline from the team, so I’m very keen for a replacement with less foot guns

Seconded. I tried hard to use Bazel in a polyglot repo because I really wanted just one builder.

Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path.

I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually usable build tool outside Big Tech. Right now many critical open source Bazel rules get a random PR every now and then from people who don't actually (have time to) care about the open source community.

My go-to now is to use mise + just to glue together build artifacts from every language's standard build tools. It's not great but at least I get to spend time on programming instead of fixing the build.

Is the reason that it works for big tech that those can spare dozens of engineers to make it work?

Yes, each of the big techs has teams that just work on the build systems, however it should also be noted that none of the big tech use the open source Bazel, Google uses Blaze internally which is what Bazel is derived from, Amazon uses Brazil which has nothing to do with Bazel and Meta uses Buck, which I know nothing of so I won't comment on it.

The major issue I found when trying to use Bazel was that its essentially a build system without specific rules for each language, hence rules support for each specific language is dependant on each language's specific community, most of which are quite tiny, and mostly maintained by upstreaming changes from their individual companies, servicing their own needs, hence a lot of work is required to make it work for your own company's needs.

Big Tech are large organizations with different needs than that of small organizations.

They care about setting standards over widely different teams and managing large-scale upgrades etc.

It's not optimized for velocity or cost efficiency, which is what a smaller organization needs.

But it doesn’t sound standardized at all if so much customization work is needed.

> The current recommendation is rules_oci, which takes the opposite approach: use only off‑the‑shelf tools...

I'm the author of one of those off the shelf tools, and the rules_oci decision here always struck me as a bit unusual. OCI is a relatively easy spec with a number of libraries that implement it. Instead of creating a custom build command that leveraged those libraries to be an efficient build tool, they found commands that could be leveraged even if image building wasn't their design.

It looks like rules_img is taking that other path with their own build command based on the go-containerregistry library. I wish them all the best with their effort.

That said, if all you need to do is add a layer to an existing base, there are tools like crane [0] and regctl [1] that do that today.

The reason other build tools typically pull the base image first is to support "RUN" build steps that execute a command inside of a container and store the filesystem changes in a new layer. If that functionality is ever added to rules_img, I expect it to have the same performance as other build tools.

[0]: https://github.com/google/go-containerregistry/blob/main/cmd...

[1]: https://regclient.org/cli/regctl/image/mod/

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching.

Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

The layers are tar files, I’m confused what behavior you actually want that isn’t supported.

The original Docker (and the current Podman) created each layer as an overlay filesystem. So each layer was essentially an ephemeral container. If a build failed, you could actually just run the last successful layer with a shell and see what's wrong.

More importantly, the layers were represented as directories on the host system. So when you wanted to run something in the final container, Docker just needed to reassemble it.

Buildkit has broken all of it. Now building is done, essentially, in a separate system, the "docker buildx" command talks with it over a socket. It transmits the context, and gets the result back as an OCI image that it then needs to unpack.

This is an entirely useless step. It also breaks caching all the time. If you build two images that differ only slightly, the host still gets two full OCI artifacts, even if two containers share most of the layers.

It looks like their Bazel infrastructure optimized it by moving caching down to the file level.

Buildkit didn't break anything here except that it each individual build step is no longer exposed as a runnable image in docker. That was unfortunate, but you can actually have buildkit run a command in that filesystem these days, and buildx now even exposes a DAP interface.

Buldkit is far more efficient than the old model.

Buildkit is still a separate system, unlike the old builder. So you get that extra step of importing the result back.

And since it's a separate system, there are also these strange limitations. For example, I can't just cache pre-built images in an NFS directory and then just push them into the Buildkit context. There's simply no command for it. Buildkit can only pull them from a registry.

> Buldkit is far more efficient than the old model.

I've yet to see it work faster than podman+buildah. And it's also just plain buggy. Caching for multi-stage and/or parallel builds has been broken since the beginning. The Docker team just ignores it and closes the bugs: https://github.com/moby/buildkit/issues/1981 https://github.com/moby/buildkit/issues/2274 https://github.com/moby/buildkit/issues/2279

I understand why. I tried to debug it, and simply getting it running under a debugger is an adventure.

So far, I found that switching to podman+podman-compose is a better solution. At least my brain is good enough to understand them completely, and contribute fixes if needed.

Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course.

I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from. You can also provide named contexts.

"Buildkit can only pull them from a registry" is just plain false.

> Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course.

I don't think that the older builder created special containers for itself?

> I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from.

Which is dog-slow, because it squirts the cache through a socket. I have an NFS disk that can be used to cache the data directly. This was just one of the attempts to make it go faster.

> You can also provide named contexts.

Which can only refer to images that are built inside this particular buildkit or are pullable from a repo.

This is really all I want, a way to quickly reused the previous state saved in some format in Github Cache, NFS, or other storage.

> I don't think that the older builder created special containers for itself?

Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd.

> Which can only refer to images that are built inside this particular buildkit or are pullable from a repo.

No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants.

> This is really all I want, a way to quickly reused the previous state saved in some format in Github Cache, NFS, or other storage.

You can cache to GitHub actions cache, S3, az blob, gcs, registries, or export to the client.

Anything you want to stick it on is going to require copying the data, and yeah that's going to be expensive.

> Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd.

No, it's not. It's a utility container that is hidden from the normal "docker ps". You can see it easily when you use docker-compose with podman.

The easiest way to see it in regular Docker is to create a simple Dockerfile with 'RUN sleep 1000' at the end and start building it. Then enter the Docker host ("docker run -it --rm --privileged --pid=host justincormack/nsenter1") and do 'mount' to see the mounts.

You'll see that buildkit will have its own overlay tree ('/var/lib/docker/buildkit/containerd-overlayfs') and the executor will have its own separate branch too. However, they do share the layers. Now wait for the container to complete building and run it.

You'll see that the running container uses an entirely _different_ set of layers. There is no reuse of layers between the buildkit and the running image.

Yes, the Docker buildkit is technically a daemon that is co-located with dockerd and just runs in its own cgroup tree. But it might as well be remote, because the resulting image runs in a completely different environment.

And the way the image is transferred from buildkit is through the containerd. Which is another separate container in the "moby" namespace.

> No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants.

Any examples?

> You can cache to GitHub actions cache, S3, az blob, gcs, registries, or export to the client.

Go on, try it. Here's a minimal repro: https://gist.github.com/Cyberax/61e6b419cd338ae7c3a7c7098abe...

First, you can build the base image, with the GHA or registry cache. It works. But the `proto` stage will never use cache. The "base" image is supplied through an additional context.

If by "utility" container you mean the containers aren't managed under the same stack, that is true. Buildkit, at least prior to docker 29, executes runc directly. It is still using the same storage backend, though there is a shim involved to convert docker's (now deprecated) graph drivers to containerd's snapshotter interface which is what Buildkit speaks. That's why there is a different tree. As of docker 29, containerd's storage is used by default. I can't recall if this used containerd to execute containers or just storage.

> > No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants. > Any examples?

--build-context foo=https://github.com/example/repo.git

Then you can "FROM foo" or whatever you want to do with that context.

> First, you can build the base image, with the GHA or registry cache. It works. But the `proto` stage will never use cache. The "base" image is supplied through an additional context.

What are you expecting to cache here? Are you saying using an extra context like this is preventing it from using the cache?

Huh?

Each layer is a tarball.

So build your tarballs (concurrently!), and then add some metadata to make an image.

From your comment elsewhere it seems maybe you are expecting the docker build paradigm of running a container and snapshotting it at various stages.

That is messy and has a number of limitations — not the least of which is cross-compilation. Reproducibility being another. But in any case, that definitely not what these rules are trying to do.

I don't quite understand how it handles running binaries then. For example, I want to do `bash -c "ls -la /"`. How would it run this command? It needs to assemble the filesystem at this point in the build process.

I guess the answer for Bazel is "don't do this"? Docker handles cross-compilation by using emulators, btw.

> “don’t do this”

Yes. The Bazel way use to produce binaries, files, directories, and then create an image “directly” from these.

Much as you would create a JAR or ZIP or DEB.

This is (1) fast (2) small and (3) more importantly reproducible. Bazel users want their builds to produce artifacts that are exactly the same, for a number of reasons. Size is also nice…do you really need ls and dozens of other executables in your containerized service?

Most Docker users don’t care about reproducibility. They’ll apt-get install and get one version today and another version tomorrow.

Good? Bad? That’s a value judgement. But Bazel users have fundamentally different objectives.

> emulators

Yeah emulators is the Docker solution for producing images of different architectures.

Since Bazel doesn’t run commands as a running container, it skips that consideration entirely.

> Size is also nice…do you really need ls and dozens of other executables in your containerized service?

Yeah, I do. For debugging mostly :(

> Most Docker users don’t care about reproducibility. They’ll apt-get install and get one version today and another version tomorrow.

Ubuntu has daily snapshots. Not great, but works reasonably well. I tried going down the Nix route, but my team (well, and also myself) struggled with it.

I'd love to have fully bit-for-bit reproducible builds, but it's too complicated with the current tooling. Especially for something like mobile iOS apps (blergh).

I went down this rabbit hole before, you have to ignore all the recommended approaches. The real solution is to have a build server with a global Docker install and a script to prune cache when the disk usage goes above a certain percentage. Cache is local and instant. Pushing and pulling cache images is an insane solution.

What you are describing is basically remote buildkitd. That allows all of your docker builds to share a big cache. The cache-to/cache-from approach is of limited usefulness.

Buildkit can be very efficient at caching, but you need to design your image build around it. Once any step encounters a cache miss, all remaining steps will too.

I'd also avoid loading the result back into the docker daemon unless you really need it there. Buildkit can output directly to a registry, or an OCI Layout, each of which will maintain the image digest and support multi-platform images (admittedly, those problems go away with the containerd storage changes happening, but it's still an additional export/import that can be skipped).

All that said, I think caching is often the wrong goal. Personally, I want reproducible builds, and those should bypass any cache to verify each step always has the same output. Also, when saving the cache, every build caches every step, even if they aren't used in future builds. As a result, for my own projects, the net result of adding a cache could be slower builds.

Instead of catching the image build steps, I think where we should be spending a lot more effort is in creating local proxies of upstream dependencies, removing the network overhead of pulling dependencies on every build. Compute intensive build steps would still be slow, but a significant number of image builds could be sped up with a proxy at the CI server level without tuning builds individually.

> Buildkit can be very efficient at caching, but you need to design your image build around it.

Well, that's what I've been trying to do. And failing, because it simply doesn't work.

> I'd also avoid loading the result back into the docker daemon unless you really need it there.

I need Docker to provide me a reproducible environment to run lints, inspections, UI tests and so on. These images are quite massive. And because caching in Docker is broken, they were getting rebuilt every time we did a push.

Well. I switched to Podman and podman-compose. Now they do get cached, and the build time is within ~1 min with the help of the GHA cache.

And yes, my deployment builds are produced without any caching.

Funny that the article only obliquely references the compression issues. The OCI users that I have seen are using gzip due to inertia, while zstd layers have been supported for a while and are radically faster.

I looked into switching to zstd recently however at least crane the utility that rules_oci uses to upload containers does not yet support uploading zstd layers.

https://github.com/google/go-containerregistry/pull/1827

Issue is over two years old. Man it is so sad how hard it can be to upstream work to big open source projects. I have a number of PRs open on both the kubernetes and etcd projects and it is almost impossible to get anyone to review them, and since nobody will review my PRs I cannot get enough work under my belt to be a committer. Sometimes I feel like if you don’t have an @redhat or @google account people just ignore you.

I think this is really close to the way nix2container works (https://github.com/nlewo/nix2container). nix2container generates metadata at build time and streams the required data at runtime.

At build time, it generates a JSON file describing the image metadata and the layers data location. At runtime, it consumes this JSON file to stream layer data and image configuration to a destination. This is implemented by adding a new transport to Skopeo. Thanks to this implementation, nix2container doesn't need to handle all various destrination since this is managed by Skopeo itself.

Recently, we introduced a mechanism to also produce such kind of JSON file for the base image (see https://github.com/nlewo/nix2container?tab=readme-ov-file#ni...).

I'm pretty sure the added (not usptreamed yet) transport could be useful in some other close contexts, such as Bazel or Guix.

I'm the nix2container author and i will be glad to discuss with you if you think this Skopeo transport could be useful for you!

(btw, your blog post is pretty well written!)

The underlying problem is that most container images are not cache efficient. Compressed tarballs arent and that’s what most of container images are. And Bazel relies heavily on caching to stay fast.

Most of the hyper scaler actually do not store container images as tarballs at scale. They usually flatten the layers and either cache the entire file system merkle tree, or breaking it down to even smaller blocks to cache them efficiently. See Alibaba Firefly Nydus, AWS Firecracker, etc… There is also various different forms of snapshotters that can lazily materialize the layers like estargz, soci, nix, etc… but none of them are widely adopted.

This is smart.

Container layers are so large that moving them around is heavy.

So defer that part for the non-hermetic push/load parts of the process, while retaining heremticity/reproducibility.

You can sort of think of it like the IO monad in Haskell…defer it all until the impure end.

Is load not hermetic? Ideally you should be mirroring all layers you use as inputs to your OCI builds and pin SHA256 versions. Your caching will also have issues if you don’t pin versions. Push should also be idempotent, but not hermetic.

Load is like a push, but with a local daemon.

> Say you have a Bazel project that builds a web application

Ok, wait, why?

Because npm, pnpm, yarn, bun, shit what else did I forget, don't support other languages. Plays well with others is a quality that they can't match.

Uhhh what? Isn’t the whole point of Bazel that it’s a monorepo with all dependencies so you don’t need effing docker just to build or run a bloody computer program?

It drives me absolute batshit insane that modern systems are incapable of either building or running computer programs without docker. Everyone should profoundly embarrassed and ashamed by this.

I’m a charlatan VR and gamedev that primarily uses Windows. But my deeply unpopular opinion is that windows is a significantly better dev environment and runtime environment because it doesn’t require all this Docker garbage. I swear that building and running programs does not actually have to be that complicated!! Linux userspace got pretty much everything related to dependencies and packages very very very wrong.

I am greatly pleased and amused that the most reliable API for gaming in Linux is Win32 via Proton. That should be a clear signal that Linux userspace has gone off the rails.

You’re converging a lot of ground here! The article is about producing container images for deployment, and have no relation to Bazels building stuff for you - if you’re not deploying as containers, you don’t need this?

On Linux vs Win32 flame warring: can you be more specific? What specifically is very very wrong with Linux packaging and dependency resolution?

> The article is about producing container images for deployment

Fair. Docker does trigger my predator drive.

I’m pretty shocked that the Bazel workflow involves downloading Docker base images from external URLs. That seems very unbazel like! That belongs in the monorepo for sure.

> What specifically is very very wrong with Linux packaging and dependency resolution?

Linux userspace for the most part is built on a pool of global shared libraries and package managers. The theory is that this is good because you can upgrade libfoo.so just once for all programs on the system.

In practice this turns into pure dependency hell. The total work around is to use Docker which completely nullifies the entire theoretic benefit.

Linux toolchains and build systems are particularly egregious at just assuming a bunch of crap is magically available in the global search path.

Docker is roughly correct in that computer programs should include their gosh darn dependencies. But it introduces so many layers of complexity that are solved by adding yet another layer. Why do I need estargz??

If you’re going to deploy with Docker then you might as well just statically link everything. You can’t always get down to a single exe. But you can typically get pretty close!

> I’m pretty shocked that the Bazel workflow involves downloading Docker base images from external URLs. That seems very unbazel like! That belongs in the monorepo for sure.

Not every dependency in Bazel requires you to "first invent the universe" locally. Lots of examples of this like toolchains, git_repository, http_archive rules and on and on. As long as they are checksum'ed (as they are in this case) so that you can still output a reproducible artifact, I don't see the problem

Everything belongs in version control imho. You should be able to clone the repo, yank the network cable, and build.

I suppose a URL with checksum is kinda sorta equivalent. But the article adds a bunch of new layers and complexity to avoid “downloading Cuda for the 4th time this week”. A whole lot of problems don’t exist if they binary blobs exist directly in the monorepo and local blob store.

It’s hard to describe the magic of a version control system that actually controls the version of all your dependencies.

Webdev is notorious for old projects being hard to compile. It should be trivial to build and run a 10+ year old project.

Making heavy use of mostly remote caches and execution was one of the original design goals of Blaze (Google's internal version) iirc in an effort to reduce build time first and foremost. So kind of the opposite of what you're suggesting. That said, fully air-gapped builds can still be achieved if you just host all those cache blobs locally.

> So kind of the opposite of what you're suggesting.

I don’t think they’re opposites. It seems orthogonal to me.

If you have a bunch of remote execution workers then ideally they sit idle on a full (shallow) clone of the repo. There should be no reason to reset between jobs. And definitely no reason to constantly refetch content.

If you did that, Bazel would work a lot better. Most of the complexity of Bazel is because it was originally basically an export of the Google internal project "Blaze," and the roughest pain points in its ergonomics were pulling in external dependencies, because that just wasn't something Google ever did. All their dependencies were vendored into their Google3 source tree.

WORKSPACE files came into being to prevent needing to do that, and now we're on MODULE files instead because they do the same things much more nicely.

That being said, Bazel will absolutely build stuff fully offline if you add the one step of running `bazel sync //...` in between cloning the repo and yanking the cable, with some caveats depending on how your toolchains are set up and of course the possibility that every mirror of your remote dependency has been deleted.

Also it is possible to air gap bazel and provide files as long as they have the same checksum offline.

Awful AI images everywhere. Can we not help ourselves?

Is my adblocker blocking them? I only saw the stack of tars in a coat. Didn't break the article's flow for me.

I also only saw that, but the text feels a bit fluffed out by AI as well, if I’m not mistaken.

It’s not. It’s been through several editing rounds. (I was one of the editors.) In theory, we don’t have a problem with AI generated content if it meets our high editorial requirements, but all Tweag technical blogs go through a rigorous, manual review and editing process to keep standards high.

As I've read through the post, seeing phrases like "Why this matters for performance", usage of em-dashes and lists/bullet points, screams AI written to me. I appreciate you saying it wasn't, but such is the fate of who wrote this to write like LLMs do nowadays. I also liked to use em-dashes and bullet lists but am consciously avoiding them now.

I interviewed a guy from Microsoft who was working on AI, and he literally speaks like this.

Like, using the words "leverage", "matters for...", "as for", and so on. And you could almost hear him doing the bullet points.

When you work with AI a lot, it changes your vocabulary.

That's absurd emdash I work with AI constantly and have noticed no such durable lexical shift.

Reply on news.ycombinator.com