Build systems a la carte: theory and practice (2020) | Hacker News Reader

Build systems a la carte: theory and practice (2020)

simon.peytonjones.org

102 pointstranspute16 comments

I have a fantasy where I win the lottery or inherit millions from a long lost relative and can live out my days developing whatever project I want. That project would be the ultimate build system.

There are at least two problems of course. I'm probably not that smart and I don't buy lottery tickets.

Maybe the long lost relative has already developed an ultimate build system nobody knows of.

I am sure that some companies would spend billions (yes, with a "b") for that. And they already have some of the smartest people on Earth working for them.

The article is more theory than practice. It practice, the big problem is not really about knowing what should be rebuilt. It is about tying the work of many people with different ideas on how thing should be done together.

The part about getting the build order correct, which is what the article is about is still important. But that's typically not the reason why everyone hates build systems. People can live with the occasional inefficiency, or "make clean". But getting that binary-only library that is not in the standard location with that source-only other library and its weird versioning scheme, both needing a different version of the same third library, written in a different programming language and using a different packet manager...

We've had `make` for north of forty years. I'm all for it being demoted to the penultimate build system but we mustn't ignore our Sisyphean tendencies.

I personally hope Nix eats the world.

This is a fantastic paper that I come back to over the years. I remember talking about it with a coworker when it came out, and we were amazed how elegantly it made the point that Excel is a build system, and how it helps understand how all build systems fit together in their design space.

An exercise for the esoteric programmer would be to make a sheet that turns Excel into an actual build system that can drive the build of a typical C program.

I'm great at excel and terrible at programming. Can you break down what's meant here?

In Excel, every cell has some number of inputs (which are other cells), and a rule for calculating the cell’s value from its input. That is basically how software build systems work, except usually they use files instead of spreadsheet cells, and they run commands (like “compile this source code”, or “combine these seven files into one program”) instead of Excel formulas.

In a build system, when you update a source code file, it should recompile the binaries that depend on it. Similarly, in excel when you update a cell in the spreadsheet, it should recompute the cells that refer to it.

The paper is about thinking about build systems in a more modular way. Are dependencies specified statically or dynamically? What algorithm is used to decide what should get rebuilt after an update? What order do we rebuild things? What happens if we pick different combinations of algorithm+order?

Is that actually a thing? What algorithms do you mean? I thought everything that depends on a change gets rebuilt.

The devil is in the details.

For example, Make decides to rebuild based on timestamp, while Bazel decides to recompute based of file hashes. Thus, if you touch a file but don't change its contents, Make might rebuild things that Bazel wouldn't. Another example in the paper is the INDIRECT function in Excel. This makes it so that Excel cannot compute the dependency graph before the build starts. Excel has ways to deal with this, but they might cause some cells to be recomputed even when they didn't actually need to be. The paper has a bunch of interesting examples like this, it's worth the read :)

The problem of dynamic dependencies is particularly tricky. For example, suppose that you want to teach your makefile that your compilation depends on the source C file, but also on all the headers that are #included by that C file. And you want to actually look inside the C file to get that list of headers, not just copy paste the list into the makefile. There's more than one way to handle this.

This seems mostly exercise in Haskell programming, and not really related to actual things that people look for in the build system.

For example, paper talks a great deal about schedulers, but those are very minute implementation details. If bazel gets "suspending" scheduler instead of "restarting" one, this is a super major change in the paper's eyes, but no user is likely to notice (other than "hey it works 2% faster now, cool!").

Moreover, the paper's scheduler abstraction is too limited - there are a whole bunch of schedulers which are non-representative in it, for example imagine a scheduler which is aware of memory and CPU usage, and schedules the tasks to avoid out-of-memory condition (like recent bazel).

The division on "cloud" vs "non-cloud" systems is also pretty weird... ok, bazel has cloud caching built-in but make requires external "ccache", but the resulting behaviors are pretty similar.

Hey, it's Dr. Simon Peyton Jones of the Haskell community! Love seeing his name pop up.

One of my favorite papers, and it guided my build system design.

I've worked on build systems quite a bit across a disparity of languages, frameworks, operating systems and engines. I've constructed some really big build systems spitting out 100's of SKUs across a dozen target platform, and I've built studio-wide build systems that again, spit out hundreds of SKU's across dozens of projects, with hundreds to thousands of programmer and artist hours per day of code and art and other binary assets pouring into the version control every hour of every day.

And the universal constant in all build systems is always a problem of management. I've turned 7 hour builds into 7 minute builds, and management doesn't care. I've saved literally thousands of hours of programmer time per year on projects, and management doesn't care. Build systems are a cost center. Always neglected, never funded, and not seen as critical infrastructure.

Build systems, at the end of the day, no matter the language or target platform, are never really that complicated, we just tend to over-complicate them. What I really want, if I had the time to implement, is a build file system.

All that said, I'll never go back to working on a big build system without having a PM that insulates me from the unreasonable demands of being woken up at 4AM by a Producer absolutely frantic because "the build system isn't working at all!" but when checking on the problem, it's because a developer checked in some code late and night and then went home. I have some serious trauma from those years of my life.

This is the first time I encountered this, and I wish I had seen it earlier. It's a great read!

Reply on news.ycombinator.com