Build systems à la carte (2018) | Hacker News Reader

Build systems à la carte (2018)

dl.acm.org

Build systems à la carte: theory and practice (revised and expanded) [pdf] - https://news.ycombinator.com/item?id=25113759 - Nov 2020 (3 comments)

Inside the Paper: Build Systems a La Carte - https://news.ycombinator.com/item?id=17494016 - July 2018 (2 comments)

Also related:

Neil who worked on this paper is working on buck2 at fb:

https://news.ycombinator.com/item?id=35472845

Code is available at https://github.com/snowleopard/build

I skimmed the paper - this is interesting on many aspects but I liked the breadth of what they analyzed... they use Excel as one of the examples!

I know it is a valuable skill to be able to efficiently break down problems into abstractions but I suspect you would still end up with "porcelain"[0] that would hide so much of this it might not be worth it. Or maybe it is so generic it becomes a standard of sorts (or certainly a standard way of doing things) with flavors of implementations on top of it?

In the conclusion they say:

> ... a simple recombination leads to a design for a monadic suspending cloud build system ...

Totally over my head as I still don't grok monads despite my belief I use them or something very close to them all the time without realizing it. I could not implement one. Anyway, I digress... neat paper and I will keep working on digesting it.

[0] https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Po...

"Monadic" here means being able to handle build steps that build build steps. The fundamental operation for a monad is `join :: Monad m => m (m a) -> m a`, which merges two layers of `m` into one. A hypothetical `Build (Build a)` (not literally in the paper but giving you the flavour) would be a build which describes how to build a build which describes how to build an a. If your build system was monadic, you could "squash" those two builds into a single one by building the inner build step, loading it somehow (maybe restarting your build system, like `make` does, or Nix's import-from-derivation feature), and then following the new build instructions to build the `a`.

Where does this arise in practice? If you had a `build :: Build x` and a function `f :: x -> Build y` that described how to compute new build steps from that `x`, then mapping `f` over `build` gives you a nested build `Build (Build y)`. This is like saying "Okay, suppose I know how to build `x`. Then since I can make a `Build y` from `x`, I can make a build that makes a build that builds `y`". If the build system is monadic, we can use `join`; this map-then-join pattern is packaged up into Haskell's "bind" operator: `(>>=) :: Monad m => m a -> (a -> m b) -> m b`, which does something equivalent to mapping the second argument over the first, and then joining the result. `(>>=)` is used in the desugaring of `do`-blocks, of which there are a few in the paper.

To illustrate "build build steps" with an example: I used to work on a project that had 1000s of Javascript files, that we compiled into a dozen or so webpack bundles to deploy. Not every webpack bundle depended on every file however. We wanted incremental builds, i.e. so that if you changed one Javascript file, you wouldn't always have to rebuild all 12 bundles (which took like half an hour) -- if only one webpack bundle included the file you changed, you should only have to rebuild that one.

You can get webpack to output a list of all the Javascript files that it used in building a particular bundle, but the list wasn't fixed from one build to the next. You could add new files or add import statements to create new dependencies. This is the situation the paper calls "dynamic dependencies".

We used "Make" to run our builds, and the way that we had incremental builds was, we had one main Makefile, and one of the tasks in the Makefile would run webpack, parse webpack's output, and literally write a bunch of sub-Makefiles to disk, describing the dependencies between webpack bundles and javascript source files. These Makefiles would then be imported into the big Makefile when a change was made and the next build was run so that Make had the dependency information and could decide whether a particular webpack bundle was stale or not.

Wanted to describe this, just in case a specific example helps you conceptualize what endgame calls "build step that builds build steps".

Incidentally, Make really sucks and this some of the buggiest, hard-to-debug, code I've ever worked with.

After reading 'Build Systems a la Carte' I decided to try reimplementing our build with Shake to capture the dynamic dependencies. The proof of concept worked like a charm and was an absolute pleasure to write, though I left the team before actually deploying it, for real. I hope the poor folks that came after aren't still wrangling Make...

Excellent example, thank you for making it concrete.

Reply on news.ycombinator.com