Ask HN: Learning about build systems generally | Hacker News Reader

Ask HN: Learning about build systems generally

19 pointsdmvdoug5 comments

The question: what resources exist to help explore build systems, specifically anything that isn’t extremely tool-specific? Or is compile/build so tool-specific that that’s all there is?

Here’s what I mean by tool-specific: resources about autotools require you to go into the guts of autotools. Resources about Cmake require you to go into the guts of Cmake. Resources about Meson require you to go into the guts of Meson. Compare that with, for example, learning about memory management. You can get quite in depth introductions and overviews of memory management without necessarily needing to get into the guts of x86-64 mm. Now, you often do. But usually as an exemplar towards the end, in order to illustrate the general principles that have been established. When I have tried to approach learning about build systems, it feels like you are immediately thrown into the weeds of how specific tools work. But my interest is not necessarily in learning one specific tool in order to use it right now.

From the beggars can be choosers front: reading material preferred, podcasts ok, videos I’m meh-ish about but if that’s what exists, hit me with them.

Good question! Here's a paper, "Build systems à la carte: Theory and practice," that compares Make, Shake, Bazel, and Excel:

https://ndmitchell.com/downloads/paper-build_systems_a_la_ca...

Yes, Excel can be considered a build system! A cell outputs the result of a computation, but it can also input computation results from other cells. For performance reasons, you don't want to recompute all cells in a spreadsheet when one cell changes. You want to do the minimum – an incremental rebuild. You do that by traversing a DAG with cells as vertices. You might not be "compiling" "files", but it sure looks like a build system in other important ways.

I've worked on build systems on & off for most of my career, but your question made me realize I've never read a book that presents build systems in the abstract. Everything I know comes from studying specific build systems and, in some cases, writing my own. The design space for build systems is huge and still expanding. That makes a unified theory of build systems difficult.

Can you share what motivated your question and what you're hoping to learn?

> Here's a paper, "Build systems à la carte: Theory and practice,"

Awesome, thank you. I actually had that saved in my files to read later, then started really digging into it after your suggestion. Almost exactly what I was looking for!

> Can you share [1] what motivated your question and [2] what you're hoping to learn?

I suppose the simplest answer is "[1] pure curiosity and [2] whatever I can to slake it." How build systems in particular piqued my interest is a weirder, zig-zaggy story involving a bunch of different things catching my eye at random, and I'm not sure that's what you wanted to know, so I'll omit for now.

obviously popular since it was submitted twice in the past month

https://news.ycombinator.com/item?id=39968887 (11 comments)

https://news.ycombinator.com/item?id=39763074 (16 comments)

Build systems may exist in the class of problems around solving constraints: trees of dependencies to resolve. Constraints can be observed in scheduling, bin-packing, memory management, and so on.

Workflows also operate under constraints: Github Actions (specific tool) can define workflows where one must wait for the other.

That brings us to managing threads. Deadlock, concurrency, and strategies to handle those are everywhere, from games to databases.

Now that we acknowledge dependencies, we have introduced state. We have state machines and state charts, reactive programming and render loops.

Build systems exist beyond code: generating a math notebook or manual page, multiple formats of an ebook, or even copying a static website to production could be a "build" (deployment).

Even so package managers and CSS. Gradle's transient dependencies, npm's peer dependencies, and CSS selector hierarchies.

So I think the one fundamental piece of these is graphs, and combining smaller data structures within nodes.

One possible exercise is creating (or importing) a graph data structure and specifying a node. The node takes input and you hook it up to outputs.

That gets into audio and shaders, dragging literal nodes around.

The rest is details: work dependent, project dependent, or interest dependent.

The basic idea of a build system is not very complex.

You have a desired output, you (in theory) know your inputs, you know the steps to take inputs and make intermediate files, and from intermediate files to final output. So just make a graph and then do the steps. There's not that much to say here.

Autotools isn't so much about building as determining the target configuration.

Many build tools try to reduce the amount of input you need to provide with regard to build steps and inputs. Some build tools try really hard to avoid unnecessary rebuilding the same source, and some try but end up not rebuilding when they should: that's mostly down to figuring out what inputs correspond to an output.

Reply on news.ycombinator.com