Programmers’ Build Errors: A Case Study at Google [pdf]

So the qualities that make Google's build infrastructure awesome are:

1. Truly incremental builds. By this I mean the build system can reliably detect when something has actually changed and exactly how much it has to rebuild. It then rebuilds only as much as it has to. There are some systems that claim they do this but almost none of them that I'm aware of other than the aforementioned Nix package manager actually do so.

1a. Googles' system does so by forcing all targets to fully specify their inputs and outputs as well as treating all targets as pure functions. No side effects allowed. The inputs determine the outputs so you know if nothing changed in the inputs then the outputs didn't change either. This means all compiles are as close to fully hermetic as you can get.

2. Distributed builds work really well. Building off the fact aforementioned guarantees for build targets. You can also build a very accurate graph of your build targets and effectively distribute all the of building.

3. Build artifact caching. Since your targets are pure functions you can safely cache them for the whole company. When you have thousands of engineers all building the same codebase frequently and the builds are highly distributed and the cache is shared by everyone the chance that someone else has already compiled a large fraction of the code you are about to compile is high.

All of this combines to make builds really fast, really reliable, and really repeatable. Which is kind of the holy grail of build systems when you think about it.

Google happens to have this because they care enough about their developers productivity that the invested thousands of man hours fixing the time sink that is compiling. That and they also build everything from head so they ran into productivity losses from build times really quickly.

The problem presumably starts when decomposing a monolithic build and how do you pull it all together again?

I have stalled at pyholodeck.mikadosoftware.com and need some inspiration to pick that over the 100 other things I needs to do :-)

One of the things in the paper was that there was a population with middling errors that did "many" builds, and a bifurcation of 2 populations that did "few" builds - one with few errors, and one with many errors. (pg 9, section 4.1 "How often do builds fail?")

Could it be that there is a group of gurus who don't need to constantly build, and which batch up testing of larger feature sets, and another group of perpetual freshmen who spend so much time fixing their fault riddled crap that they simply cannot submit very many builds? Yes, I know they attributed at least some of this to the employee role being done, but it makes me wonder. "Experience" would not tease this pattern out. (while it seems to me customary to build more often in the initial stages of using a new language, we've all worked with "that guy" who just bumbles through, year after year)