Reproducibility in Disguise | Hacker News Reader

Reproducibility in Disguise

fzakaria.com

54 pointssetheron24 comments

One other hidden Bazel trap that I've seen is for companies to migrate a large codebase to Bazel, but then to rely on OS-provided tools and libraries. Commonly, this gets paired with a glib answer like "it's fine, we build from inside of a Docker container". But I've never seen that Docker image linked into Bazel's dependency resolver, or the compose scripts used to launch the container.

This has the following effects:

    1. There are unexpressed package/tool dependencies.
    2. Across a large organization, Bazel's reproducibility guarantees go out the window.
    3. Developers can't just clone the repo and start using Bazel. Instead, they have to pull down some pinned Docker image, or build it themselves and lose reproducibility.
    4. This effectively poisons the cache whenever the Docker image is updated or rebuilt. If using a shared remote cache, it can be a major issue.

If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem?

Docker did us wonders but also set us back in many ways (same view point about the proliferation of development on MacOS).

I've lately been thinking why isn't there a public vendored third_party set to use.

If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win.

> If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win.

“It has been 0 days since a minor version update in a third-party library has broken my code.”

This is just a fiendishly difficult problem to solve.

> If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem?

Here are some ways you can approach this problem:

If you have a bunch of shared tools & libraries and you don’t want to vendor the whole packages, you can vendor the artifacts. You can do this piecemeal or as a big chunk. Basically, instead of a Docker image containing your compiler or whatever, you have a tarball. You make Bazel responsible for downloading the tarball. Or maybe it’s several tarballs, it doesn’t matter. Bazel is actually pretty good at this.

Another approach is to combine Bazel with something else reproducible, like Nix. For now, I’d suggest taking a very basic approach, the most simple approach, which is to create an environment using Nix (containing your toolchain, third-party libraries, and Bazel) and then building your code inside that environment using Bazel. You can use the local_repository() rule to access the Nix environment.

Bazel + Nix is a really good idea, but maybe now is not the time to dive into that. When I’ve investigated the Bazel + Nix combination, it looks like neither system is exactly stable / mature enough yet. Bazel just recently launched bzlmod and the Bazel ecosystem is shifting to use bzlmod everywhere. Nix is shifting to flakes but there are some pains there too (especially around documentation, but there are also some things that don’t quite work with flakes yet). So in the future, you could do something kind of, well, cursed, where you have a Bazel and Nix lasagna. You use Nix to run Bazel, and then you use Nix packages as dependencies of your Bazel build. In theory, this could give you the best of both worlds—you get the wonderful fine-grained dependencies of Bazel, the rich build system, the super fast performance. And then you get access to reproducible builds of all sorts of third-party dependencies through Nix. In practice, you need to become something of an expert in both Bazel and Nix in order to do this.

As a final option, depending on the languages you are using, you may be happy with just plain bzlmod. Like, Go integration with Bazel is damn solid. You don’t need to vendor anything, you can just keep using go.mod, and Bazel will deal with it (with some help from Gazelle). Bazel will even download the Go compiler for you, without any additional setup. A lot of other languages work the same way—it’s just that C and C++ are notable exceptions, where Bazel just grabs the system compiler by default, and package management for C and C++ is complete chaos.

Is there any good way to make some Bazel targets hermetic, and some non-hermetic? I'd love to be able to set my build up with something kind of like:

    1. Non-isolated Bazel target that produces a container image.
    2. All other targets are isolated, and build inside of a container created from the new image.

This is a good question, but it took me a few tries to write a satisfactory answer.

A hermetic build is one where the environment can change but the build outputs stay the same. This lets you aggressively and pervasively use remote caches, which can give massive performance benefits to large and complicated builds. I’m sure you know this, I’m just going through so I can clarify it in my mind.

If your Docker image depends on the environment but your final build does not, then congratulations, you’ve made a hermetic build. But I don’t see how building the Docker image in Bazel is helpful here. The more likely result here, I think, is that you have a build which is still non-hermetic. The inner build steps are isolated from your current environment, but they are not isolated from the environment used to create the Docker image.

As a matter of practical advice, try creating your Docker image in Nix. Nix is very good at creating Docker images hermetically and has a large number of third-party packages and toolchains available. Bazel is very good at fast builds and fine-grained dependencies. The easiest, simplest thing you can do with this combination of tools is to have Nix create a Docker image that contains Bazel and all of your build dependencies that aren’t vendored into the Bazel tree.

You can create hermetic builds by carefully designing your build rules so they aren’t affected by the environment. But it is a lot easier and more reliable to just stamp SHA-256 checksums on chunks of your environment, when that option is available.

I will say that it's _extremely easy_ to just list `Dockerfile` in your output dependencies if it matters enough for you to care.

Stuff from apt/etc as well? put the install script in the dependencies.

Really care about the version numbers? Add a test that just calls `tool -v` and compares the output to some fixed number. Add a big comment.

Is it perfect? No! Is it hermetic! Hell no! But is it going to catch a hell of a lot of stuff and get you fast tests along the way? Yes!

The biggest point with all CI/CD is to get as much value as you can from these systems, while introducing the least amount of friction. And velocity gained from these systems mean that if you find an issue, you can fix it quickly. It's an amazing feedback loop, and leaning into that feedback loop is way more valuable than writing `gettext` compilation scripts.

But what does installing things via apt have to do with reproducibility? Does apt have some way to specify lock files or hashsums of the packages it is supposed to install? Or do you mean to pin version numbers of system packages and rely on that? Otherwise apt would be the point where the guarantees go out of the window already.

Debian can pin packages to certain versions by their numbers (see dpkg(1), '--set-selections') and it does verify package integrity. I can't think of any way to pin a package to a hash like with Bazel or Nix, but the expectation is that packages are not changed after publication in dpkg repositories - and for Debian itself, that expectation is a strictly-followed rule.

Therefore I would trust package pinning to work, but it's not quite as straightforward for the end-user as unique package hashes as identifiers.

I'm guessing this problem might be language-specific. Does Bazel do better at importing external dependencies for Go?

You'd have the same problem with CGO or JNI for Java. C/C++ shared libraries are never part of the package versioning in language package managers (that's the part we stick our heads in the sand about)

If you look at how cgo is linked, you'll see it's a magic comment in go source code: https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80...

If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues.

Admittedly, you'll usually get compilation errors for undefined references in compiled languages (modulo dlopen), so at least you'll get a build error instead of a runtime error like you do in python.

The solution for both Go and Python is for bazel to also control the system dependencies, i.e. to force you to run all code in a container sandbox bazel controls, force you to specify all external dependencies from linux kernel version up to go compiler version, and build all of that itself.

In other words, bazel is a lacking implementation of NixOS.

> If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues.

That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry.

Go has an internal build system as well as a compiler. The build system drives the compiler. When you use Bazel with Go, you’re bypassing the Go build system entirely. Bazel directly invokes the Go compiler. This means that any of those comments will also get ignored. (Generally, this is how Bazel works for other languages too. Bazel + Rust does not use Cargo. Bazel + Java does not use, like, Maven or Gradle.)

You have to specify the dependencies in your build file. Let's say you have a Go library abc, with a C library xyz.

  go_library(
    name = "abc",
    srcs = [
      "abc.go",
    ],
    cdeps = [
      "//path/to/xyz",
    ],
    cgo = True,
    importpath = "path/to/abc",
  )

> In other words, bazel is a lacking implementation of NixOS.

Bazel and NixOS are both good tools for making reproducible builds.

They’re even a good match for each other—grab your compiler from NixPkgs, and build the rest of your project in Bazel. You get the benefit of Bazel’s fine-grained dependencies (Nix has very coarse dependency management), and you can get some specific version of GCC built for both Linux and macOS by tapping into NixPkgs. A marriage made in heaven. (Except for the fact that you are now using two tools which both have very steep learning curves.)

There are a lot of interesting similarities between Bazel and Nix, which is not surprising, since they are solving similar problems.

> That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry.

> This means that any of those comments will also get ignored

I cloned this example: https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea6...

And then ran "go get github.com/xthexder/go-jack" and added an 'import _ "github.com/xthexder/go-jack"' to 'cmd/roll.go'.

    $ bazel run //:gazelle-update-repos
    $ bazel run //:basic-gazelle roll
    __main__/external/com_github_xthexder_go_jack/jack.go:11:10: fatal error: jack/jack.h: No such file or directory
    compilation terminated.
    compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/cgo: exit status 2

    $ apt-get install libjack-dev
    $ bazel run //:basic-gazelle roll
    Number rolled: 80

Sure looks like you're wrong. Bazel obviously didn't ignore those comments because it errored out on trying to find the include. (And of course it didn't ignore them, those comments are preprocessor instructions for the go compiler, which bazel runs under the hood). It obviously didn't need me to specify cdeps because it found the C dependency on my host when I installed it without changing a single bazel-related file.

It looks like your understanding might be wrong, sorry.

You’re making a complaint about Gazelle. If you import third-party packages with c dependencies with Gazelle, obviously, the two choices are it breaks hermeticity or you vendor the C code.

Bazel does ignore those comments. Gazelle does not.

> Bazel does ignore those comments. Gazelle does not.

I really don't understand what you're saying.

gazelle does not seem to do anything related to the c dependencies there. The full diff gazelle generated is just changing "deps" in 'BUILD.bazel' files to include the library, and adding a "go_repository" to "deps.bzl". Not that it would matter anyway, gazelle is part of bazel.

There's no references to c dependencies anywhere in what gazelle produced. I could have hand-written those changes just as easily, without gazelle.

I'm only using gazelle here because that's the only go example that upstream has in the repo.

The "bazel build" part is very clearly the part that is looking for these C files, and doing so in a non-hermetic way.

It's possible to make it hermetic, sure, if you pay attention and vendor things or such, but it's obviously not required.

That's all I was saying, just like you can have python break hermeticity and depend on external C libraries, you can have go break hermeticity and depend on external c libraries. It's possible to use bazel such that it's hermetic, but it doesn't stop you from doing it wrong.

That's all this thread is about, is whether it's also possible to make these non-hermetic references in languages other than python.

Like, if you go back and read my commend and your comment a few up, you'll see that my original claim was just that you could impurely reference the host "libjack-dev", which I have very clearly shown you can, and your claim was that it's literally impossible to do that, which it very obviously is possible.

> There's no references to c dependencies anywhere in what gazelle produced. I could have hand-written those changes just as easily, without gazelle.

I get that this scenario is confusing but this is incorrect. The problem is that you are running Gazelle to generate the build files for third-party dependencies, and not just the build files in your own workspace.

  $ cat bazel-basic-gazelle/external/com_github_xthexder_go_jack/BUILD.bazel
  load("@io_bazel_rules_go//go:def.bzl", "go_library")

  go_library(
    ...
    cgo = True,
    clinkopts = ... "-ljack" ...,
    ...
  )

This file is generated by Gazelle. Gazelle is the tool you are running when you run this command:

  bazel run //:gazelle-update-repos

When you run that command, it generates go_repository() rules, which are a part of Gazelle. There are three things at play here—there’s Bazel, rules_go, and Gazelle.

Bazel by itself doesn’t have any Go rules. You need to use rules_go for that. The rules_go rules completely ignore any of those comments you’re talking about—all libraries you link in are specified by things like cdeps and clinkopts.

Gazelle is what fills those in, and go_repository() is a part of Gazelle. You can see that in the imports:

  # This is part of @bazel_gazelle, not @io_bazel_rules_go.
  load("@bazel_gazelle//:deps.bzl", "go_repository")

There are a few practical suggestions I have for dealing with this:

- Nix. Run Bazel inside a Nix environment that contains your C dependencies. (You can have Bazel bring in C dependencies using Nix, too, if you can set aside multiple weeks to figure out how to do that. I wouldn’t.)

- Make your Bazel more hermetic. It’s not completely hermetic out of the box, but you can configure it to be more strict.

- For pure Go projects, consider disabling CGO with --@io_bazel_rules_go//go/config:pure.

- For third-party dependencies which use CGO dependencies, vendor those dependencies. You don’t need to vendor everything, just the dependencies that use CGO.

My own personal experience is that Bazel has a damn steep learning curve, which is why I don’t like engaging in Bazel apologetics. Bazel is reproducible when used/configured correctly, and it’s way easier to use/configure Bazel to do that than it is to configure other systems—but it’s still a pain in the ass.

And my own personal experience with the Go ecosystem is that CGO usage is uncommon enough that, most of the time, I can get work done with CGO turned off. YMMV.

Okay, yup, you're right, the C dependency stuff is gazelle.

I'll still claim the meat of what I said was correct:

> If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues.

> Admittedly, you'll usually get compilation errors for undefined references in compiled languages (modulo dlopen)

That's the important bit of my comment, and it indeed seems true that bazel, as commonly used for go with gazelle, does allow and even facilitate those impure references to host dependencies, so even though I didn't understand the details of it fully, my overall point seems entirely accurate.

I appreciate you taking the time to explain things in more detail and share some advice/wisdom on dealing with this!

Agreed. It feels like shared dependencies are akin to globals in your language of choice. Sharing dependency instances among components creates problems. A uses b and c, both of which want to use different versions of d. The simple answer, which is understandably difficult for existing languages to solve, is to just keep 2 copies of d.

Go does keep both versions like that for major versions (which are treated like independent libraries), but not for smaller upgrades that are supposed to be compatible.

edit: The previous title was "Reproducibility in Disguise: Bazel, Dependencies, and the Versioning Lie"

Eh... the real lie is the signal version policy in the first place. It can be broken, at least in Google, and must be often otherwise there'd be massive gridlock.

If you want to bring in a new third-party library that has a dependency that already exists in the monorepo, you'd better _pray_ that the version in the monorepo is already compatible. If not, you have to update that library and potentially every target that depends on it, maybe transitively. Trying to do that can sometimes take years - no joke.

So the single-version policy flexes, and exceptions are supposed to be temporary, but they're not always.

In the end there are two competing sometimes good, sometimes bad goals:

- Libraries should be somewhat discouraged from making too many or too flippant breaking changes. By having to update clients, they feel the pain and take migration more into account.

- Libraries should be able to make real improvements, even if they are breaking changes, and having clients shouldn't burden them with unbounded costs. They should be able to distribute reasonable costs of updating by letting clients update on their own time with versioning.

Vendored third party packages only highlight this tension, because the external world has largely settled on versioning as the approach which clearly conflicts with signle-versions, but it really existed anyway.

Single version policy doesn't solve version conflicts but it does put it right in your face which is as good as we can do.

Without single versions, new packages to the SAT solver can introduce new duplicate shared objects secretly breaking your hermetic seal.

It doesn't hide the complexity like package managers do by picking a language package by SemVer and crossing our fingers that the shared objects included will work too.

The gross hack is you import libbfoo1 and libfoo2 when breaking changes like that are involved

Then the symbols interpose (shadow) and now you have divergent behavior depending on the order they are loaded (which can change in Python as they are dlopen)

Reply on news.ycombinator.com