Your Version Control and Build Systems Don't Scale

A friend of mine expressed the problem described here in a more formal manner in this paper: http://gittup.org/tup/build_system_rules_and_algorithms.pdf

He ended up creating his own build system, "tup" < http://gittup.org/tup/ >, based off of it. It also has the property desired in this article that, "No-op builds should be O(1) and instantaneous, and most other builds should be O(WhateverChanged)".

I strongly disagree with the assertion that C compilation is I/O bound. If you have optimizations turned off, this is probably true, but if you are using a decent optimizing compiler, this is not the case.

We ran some tests where I work on a pretty good sized code-base and found that we were CPU bound, even on an 8-core system.

[edit] There is an exception, if you are using networked file systems of some type (especially clearcase dynamic views) then you are almost certainly I/O bound.

This is the one thing I miss from working with Java in Eclipse. There, you're always in a built state so every time you save the file you're working on, you instantly get all the compiler errors if any. There's also a large amount of errors the editor can detect without invoking the compiler, and you get constant feedback about those as well.

My current main environment is Visual Studio for C#, and there you get a lot of errors detected while editing, but not all of them, so you have to continually press the rebuild button, and the time of that rebuild just keeps growing...

Then again, I recently bought a SSD, that helps a bit with build times. :-)

Good article about large builds. I know of at least one outfit with a large legacy C++ code spend the effort to move to C#, in no small part due to unmanageable build times. This would be under the "constant reduce" leg of this journey, but with C# you cut your file count in half all other things being equal.

A minor nit: most other builds should be O(WhateverChanged) -- consider a C++ header file used by 10 c++ files. Those 10 would need to be recompiled. The agony there is that you wish for some less-than-full-file dependency analysis so that only the files that use that constant you just changed need to recompile.

On the whole a good start to those massive builds you have.

Here's a mirror (don't know how well it will hold up): http://hildr.luminance.org/ibb/

The idea of optimizing build performance for huge monolithic chunks of code seems somewhat misguided. Isn't it better to break your product up into a set of reasonably sized libraries that can be built separately? That way when you change something in one module you can just rebuild that one library, and then perhaps relink it with the others. That general approach can usually deliver fast development cycles with any language or build tool.

I like the idea of in-memory change detection... there's a definite use there. I don't agree with the idea of storing all files in memory (especially given their complaint that Git is slow on 20GB of files...), though I admit grepping that is fast.

But what if it were simplified to just a live ls-diff based on when a command was last run / a timestamp? If Git / Make would hook into something like that it'd be really useful.

I agree with the general thrust of the article in that iteration time is one of the most important factors of any kind of creative work, programming included. The longer you go without feedback or corrections, the less confident you can be that you're on the "right track."

That said, build systems and VCS aren't the only ways to optimize the process. For example, a system using scripts processed at runtime lets you reduce the amount of compiled code and iterate quickly on the one section by pressing a "reload script" button, or, even more conveniently, monitoring the source file and auto-reloading when changes are saved. If you're daring enough you could probably even do this with machine-compiled languages by abusing dynamic linking, though the ease of doing that would ultimately depend on the runtime linker capabilities.

Doing this kind of data-driven thing will cost runtime performance since less can be compiled and optimized statically, but if you treat it as the "scaffolding" work that it is and also include ways to reclaim some of the static factors for release builds, you can get a better overall result than you would if you were just suffering through long builds.

Whilst I hated clearcase as a version control system (and I mean 'fire and stakes' hate, not just 'crossO the road') - it did have one good trick up its sleeve:

Since the dynamic views into the source were implemented as a custom file system, they could track all the file access that went into building each object. They supplied a custom version of make that pulled that info. out to construct complete dependency descriptions. If you had the toolchain in clearcase as well, that would include all the compiler bins, libs and includes.

Once that info was cached, and as they had code at filesystem level, it could easily be invalidated as files were touched.

Another product of this mechanism was that if you tried to build something that someone else had already (all the same inputs), it would just grab it over the network. In practice this meant that the nightly build would cache the bulk of the object files.

FWIW: I'm not sure I buy that 21 minute Mozilla build, unless he was working on a cold disk, or unless the build is doing a bunch of stuff with broken dependencies. If the build was just done, the file metadata is in the page cache, and you can read a staggering number of dents in 21 minutes.

Disable make’s implicit rules with -r or explicitly in the Makefile:

    .SUFFIXES:
    %:: %,v
    %:: RCS/%
    %:: RCS/%,v
    %:: s.%
    %:: SCCS/s.%
    %.c: %.w %.ch

This gave me a significant speedup for no-op builds (0.7s → 0.1s, ~500 goals in the Makefile).

Of course you can only do this if you do not rely on these implicit rules.

My version control and build systems scale better than your blog.