Comment by keithwinstein | Hacker News Reader

Comment by keithwinstein | Hacker News Reader

Parent Full thread

I've been confused for a while about the common claim that Bazel gives "full hermeticity" of builds -- it doesn't seem to be true in practice (at least for packages with system dependencies). Maybe you can help me clear it up.

E.g. Google's protobuf libraries [1] can be built with Bazel, and they depend heavily on system headers outside the repository, e.g. <iostream> and <stdio.h> and lots more. If those headers subsequently change, Bazel will not pick up on this and will not know to rebuild the parts of the build that depend on them.

To reproduce: run `bazel build -c opt //:protoc_lib` and then put random garbage in your /usr/include/stdio.h and /usr/include/c++/<version>/iostream and then rerun the bazel command -- it will not know to invalidate the build cache. If you `bazel clean` and then build again, you'll get different results.

Bazel does a lot of really nice things and I can believe that within a google3-like environment (where the source code never references a system header?), it effectively provides hermetic builds, but in practice as used outside Google (or even in Google's public OSS releases) it doesn't seem to really match this description or enforce a hermetic seal. What am I missing?

[1] https://github.com/protocolbuffers/protobuf

The best way to get your hermetic builds in order is to _always_ use remote execution and ensure that your executor nodes don't have compilers and headers just laying around. Force yourself to get the toolchain under bazel's control.

I think the reason the public finds this so mysterious is the documentation for CROSSTOOL is terrible, and virtually all of the people who learned how to use blaze at Google go out in the industry with no understanding at all of CROSSTOOL, because there's a small dedicated team who maintain it.

Does your company use remote execution across the board with Bazel? How much effort was it?

I work for no man, as a character in a movie once said. But I recently went through this exercise just to make sure I could do it, and it's easy enough to put a bare-bones executor node in the cloud and use it temporarily. You can even get ARM nodes, to make sure you aren't implying the target architecture.

Doing it on a company scale is probably harder. Last place I worked used bazel but did not bother with remote.

I wish Bazel was more explicit about things leaking in from outside the workspace, but you can always vendor the toolchain to be more hermetic. For example see https://github.com/grailbio/bazel-toolchain which sets up LLVM and Clang for you.

https://github.com/bazelbuild/bazel/issues/4558

Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf. (And not particular to Bazel -- a developer can make any build system effectively hermetic by vendoring everything.)

In my view the challenge here is that a dependency changes (e.g. /usr/include/stdio.h is upgraded by the system package manager, or two users sharing a cache have different versions of a system library) and Bazel doesn't realize that it needs to rebuild. It would be a pretty heavy hammer if the way to fix that requires every OSS project to include the whole user-space OS (C++ compiler, system headers, libraries) in the repo or via submodule and then be careful that no include path or any part of the build system accidentally references any header or library outside the repository.

And maybe this issue just doesn't need to be fixed (it's not like automake produces build rules that explicitly depend on system headers either!) -- my quibble was with the notion that Bazel, unlike CMake or whatever, provides fully hermetic builds, or tracks dependencies carefully enough to provide an organization-wide build cache across diversely configured/upgraded systems.

> Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf.

No, that's up to the user. If you include protobuf in your Bazel workspace and have a toolchain configured, protobuf will be built with that toolchain (this is also how you would cross-compile). Bazel workspaces are still a little rough around the edges and interactions between them can be very confusing, as Google doesn't need them internally (everything is vendored as part of the same tree).

Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.

Of course, the whole model falls apart if you want to depend on things outside of Bazel's control. However in practice I've found that writing Bazel build files for third-party dependencies isn't as hard as it seems, and provides tons of benefits as parent mentioned.

For example, last week I was tracking down memory corruption in an external dependency. With a simple `bazel run --config asan -c dbg //foo`, I had foo and all of its transitive dependencies re-compiled and running under AddressSanitizer within minutes. How long would that take with other build systems? Probably a solid week.

Reply on news.ycombinator.com