The next generation of Bazel builds

bazel has arrived at $WORK and it has been a non-trivial amount of work for even the passionate advocates of bazel. I know it was written by the Very Smart People at google. They are clearly smarter than me so I must be the dummy. Especially since I never passed their interview tests. :-)

Of course given all things google, by the time I'm fully onboard the train, the cool kids will be making a new train and then I'll have to hop onto that way to enjoy the rewards of the promised land that never quite seem to arrive.

Almost all of the distinctions you mentioned are related to the way that Bazel has the concept of a "target", which lets the build graph work at a higher level than individual files.

Suppose you write the following in a BUILD file:

    cc_library(
        name = "foo",
        srcs = ["foo.c"],
        hdrs = ["foo.h"],
    )

    cc_library(
        name = "bar",
        srcs = ["bar.c"],
        hdrs = ["bar.h"],
        deps = [":foo"],
    )

It's nice to be able to query the build graph at this high level. It lets you see the relationship between components in the abstract, rather than a file-by-file level. That is what "bazel query" does.

But sometimes you might want to dig deeper into the specific commands (actions) that will be executed when you build a target. That is what "bazel aquery" is for.

Macros vs. rules is basically a question of whether the build logic runs before or after the target graph is built. A macro lets you declare a bit of logic where something that looks like a target will actually expand into multiple targets (or have the attributes munged a bit). It is expanded before the target graph is built, so you won't see it in the output of "bazel query."

If you took away the target graph, I think you'd take away a lot of what makes Bazel powerful. A key idea behind Bazel is to encapsulate build logic, so that you can use a rule like cc_library() without having to know how it's implemented or exactly what actions will run.

I don't say this to minimize any of the pain people experience when adopting Bazel. I'm actually curious to learn more about what the biggest pain points are that make it difficult to adopt.

So in my mind, Bazel is no longer worth it unless the savings are so great that you can afford to staff a build team to figure these things out. Most teams would benefit out of using simple command runners instead of fully-fledged build systems.

Compared with build efforts using other tools and non-monorepo setups at other companies the effort here has felt much reduced.

The reason a lot of people like Bazel is, I think, tools like Gazelle -- which reduce that whole problem back to "Run gazelle" and all the crap is taken care of for you. Dependencies, BUILD files, etc. People constantly talk about the "complexity" aspect, but very few people appreciate how complex Cargo, NPM, Yarn, Cabal, Dune, internally are. Because they just run "build", and it works. Bazel, Buck2, Mill, etc will all have this problem unless huge effort is put in.

TBH, this is one of the reasons why I think Nix has wildly succeeded in the past few years while more fine-grained and scalable systems have had adoption problems -- despite its numerous, numerous flaws. You get to Bring-Your-Own-Build-System, and Nix along with the blood of 10,000 upstream contributors keeps the juice flowing, and it's cached and hermetic so you see real savings. That greatly eases people into it. So they adopt it at all points on the curve (small, medium, huge projects), because it works with what they have at all those points. That makes them willing to get deeper and use the tool more.

https://registry.bazel.build/all-modules

But with Buck2 you're stuck with `http_archive` and vendoring.

Without commenting on the above statement:

> For example, Bazel throws away your analysis cache when you change flags that have nothing to do with what's being built or how, like flags that change what tests are run.

I don't think this is a good example. Bazel's analysis cache clearing is to preserve correctness in the context of a legitimately algorithmically difficult problem. The fact Bazel has this limitation is a testament toward its correctness strengths. I'm not aware of systems that have solved that algorithmic problem but curious if anyone knows any.

Also Bazel avoids that problem for most "flags that change which tests are run", since that subset of the problem is more solvable. --test_env was a notable exception, which was fixed (https://github.com/fmeum/bazel/commit/eb494194c1c466f7fd7355...) but not sure if it's in the latest Bazel release yet? But generally changing --test* has much smaller performance impact.

- Cross-compilation and target platform information

- Fetching dependencies

- Toolchains

I'm not sure a system that solves these would still be considered minimal by most, but those are table-stakes features in my view.

If you don't need these things, maybe stick with Make?

I'm still unsure if a complete, viable eden solution was released with all of its necessary components.

That has some pretty useful properties for a config language that I'm not sure TypeScript helps much on. That said, Lean also has these properties too as an alternate language with good IDE support, and it would be pretty fun to write proofs of your build scripts.

> a lot of performance gotchas

Can you list some examples?

I think the part I hated most about Bazel was the fact that there were in fact two dialects; one for BUILD files and one for .bzl files. The things you do in them are different but the documentation is always vague on what goes where.

So GPT has likely post-trained on a lot of Google code knowledge.

Now consider that you are editing 50 lines of source for a tool which will then need to be executed on some particular platform to generate some parts of your project.

Now consider that you are editing 50 lines defining the structure and dependency graph of your project itself.

[0] https://gittup.org/tup/ex_a_first_tupfile.html

[1] https://github.com/gittup/tup/issues/113

Back to bazel, I am of the impression that some of its complexity comes from a requirement to handle heterogeneous build systems. For example, some python development requires resolving both python dependencies and C. Being good at either is a bunch of work; but, handling both means rolling your own polyglot system or coordinating both as a second class citizen.

It is not at all uncommon to have changes percolate out into larger impacts than you'd expect, though. Especially in projects that attempt whole program optimizations as part of the build.

Consider anything that basically builds a program that is used at build time. Which is not that uncommon when you consider that ML models have grown significantly. Change that tool, and suddenly you have to rebuild the entire project if you didn't split it out into a separate graph. (I say ML, but really any simple linter/whatever is the same here.)

...and the transitive closure of those parts, which is where things get complicated. It may be that the output didn't actually change, so you can prune the graph there with some smarts. It may be that the thing changed was a tool used in many other rules.

And you have to know the complete set of outputs and inputs of every action.

And on and on.

• Adding a file. It hasn't been read before, so no tasks in your graph know about it. If you can intercept and record what file patterns a build tool is looking for it helps, but you can't easily know that because programs often do matching against directory contents themselves, not in a way you can intercept.

• File changes that yield no-op changes, e.g. editing a comment in a core utility shouldn't recompile the entire project. More subtly, editing method bodies in a Java program doesn't require the users to be recompiled, but editing class definitions or exposed method prototypes does.

• "Building" test cases.

• You don't want to repeat work that has been done before, so you want to cache it (e.g. switching branches back and forth shouldn't rebuild everything).