Build System Schism: The Curse of Meta Build Systems
yzena.com
yzena.com
I have never enjoyed the experience of working with tools that did this. How does it know which files I actually want, and which files are there because I (or somebody else) got something wrong? The answer, of course: it doesn't.
I moan a lot about having to do the computer's work for it, but I think in this case I am happy to put up with some minor drudgery (adding a file name to a list, once) in exchange for the computer's boundless ability to then patiently follow my instructions without trying to second-guess me on every single build.
Yes, I want my build fast. However, give me a way to kick off a scan to look for extraneous stuff.
why would you put a file "foo.c" into directory "src/" if you don't want to have it compiled? put it somewhere else then.
Rig will do this too. You can statically list dependencies.
If turing-completeness is necessary, why should there even exist a build system independent from a programming language now?
Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from?
So when you build Rig, build it in C++ and make C++ your scripting ... Ok, that left a bad taste in my mouth.
Seriously, though. Build systems as the author describes them are going to compete against the project managers that come with languages (Cargo, Go). The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C.
While I applaud the author for the attempt to improve this ecosystem, I really hope it will become less relevant some day.
I want my build system to be independent of the programming language for two reasons:
1) What gavinhoward said: I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. Much like I want to use the same IDE regardless of language.
2) I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers.
For me, the separation of the two roles is very important. If they were tied together, it would mean that I'd have to add complexity elsewhere to cover my use cases. Languages that include their own package managers (like Rust, Go, etc.) give me headaches.
It’s not obvious to me that one can solve the problem of making N languages co-operate by adding another. Why not just use the most powerful language to orchestrate everything?
> I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers.
That begs the question of whether compiler upgrades should force build system upgrades. In a mature programming language, this should never be the case, although practically speaking it’s definitely possible (e.g. ASDF 3.3, released earlier this year but built on a language which has been static for thirty years, adds compatibility for Allegro Common Lisp!).
Incidentally, ASDF is pretty cool: https://asdf.common-lisp.dev/
I usually do everything in a single language, but sometimes that's not the best solution. It's not a matter of how powerful the language is in a general sense, but how good it is at a particular kind of task.
But even ignoring my projects that involve multiple languages simultaneously, I still regularly use several languages in total, even if only one per project. So having a build system that's independent of the language is still highly desirable.
> That begs the question of whether compiler upgrades should force build system upgrades.
My answer to that is "no, never", of course. That leads to increased complexity for me, in exchange for no gain. The build system should have no special knowledge or expectation of what language or language tools are being used.
Ah, that’s a difference between us: I prefer to always use one primary language, and all others are secondary languages selected per-project. So it makes sense for me to just use a system built in my primary choice.
> My answer to that is "no, never", of course.
Agreed, too! In principle upgrading the compiler should have no effect whatsoever on correct, portable programs. OTOH, there sure do seem to be an awful lot of incorrect and unportable programs out there!
Rails
Bundler
Vagrant
Chef
Opsworks configuration
JavaScript bundling
Generating PDFs
Probably other things
It was so bad when I started that I couldn't even get the system up and running locally. I ended up tossing hundreds of lines of random ruby code and with a few lines of docker commands had it up and running.
Since then I've preferred keeping things nicely not dependent on each other whenever possible.
This is a great question that I will make sure to answer in my Turing-completeness post in a week.
But to answer now, it's because of multi-project builds, where projects can be in multiple languages.
If you have build systems that are not independent from their language, how do you get them to play nice?
I designed Rig to be able to act as a babysitter and make them play nice.
If I play my cards right, Rig will be the reason other build systems become irrelevant. :)
EDIT: s/Skylark/Starlark/
The thing about Starlark is that it is effectively Turing complete.
You can simulate a while loop with a long enough foreach, an if statement, and a break statement.
And by "long enough," I mean long enough to make anyone just stop the build, but hey, if I stack the foreach loops, I can make the inner loop exponentially long, even though ints are limited to 32 bits.
By not taking out the break keyword, Starlark remains Turing-complete.
And yes, I tested this.
A primitive recursive language gets you most of the benefits of a general recursive one _without_ needing to be _actually_ Turing complete. So this isn't "bad news" for me. I'm saying that I think you should avoid using a general recursive (i.e. Turing complete) language for a build system and instead have guarantees that the system will terminate by using a primitive recursive language of some kind because you literally, as you did, need to go out of your way to get code written in such a fashion to run for an inordinate amount of time. And yet, an inordinate amount of time isn't forever and unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
If it's only guaranteed to terminate after the heat death of the universe, it won't terminate, so I would call that "forever."
> unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
And this is exactly why I said that break was necessary.
Sure your static analysis may detect that the loop goes 2^256 times, which would be "forever," and so it returns the answer, "Does not halt."
Ah, but with the break keyword, it still could. In fact, it might always halt rather quickly.
Rule of thumb is that if static analysis can be wrong, it is Turing-complete.
https://github.com/aappleby/Hancho
In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.
End users get to pick a familiar language to them.
Similar to how Pulumi is available in multiple languages.
Then code both builds and meta build concerns as you would code anything else.
Please no. I don't want to handle filename and other string manipulation in C.
What about packages for *nix based OS's? dnf, pacman, apt, etc. essential have there own "meta" build systems to create packages for there particular package flavor weather that be rpm, deb, tar.gz etc.
C/C++ are big time numerous and serious real. And they need all the help they can get to do builds more like golang
It shouldn’t. The only issue is that most folks aren’t using powerful enough languages. As you note, using C++ as a scripting language would be pretty crazy. So don’t do that!
Anyways, I’m looking forward to more options in this space. The current pattern of rebuilding everything from scratch every time in a non deterministic way is the gross reality of GitHub Actions and Dockerfiles. It would be great to have more people working on a more cacheable build system.
That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
But I can email you directly if you wish.
> It would be great to have more people working on a more cacheable build system.
Rig is not that yet, unfortunately. I don't want to get your hopes up.
It can't even do incremental builds yet. And while it has multi-threading, I haven't tested it.
This is very alpha software.
> That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
Generating those files was my plan, but friction is not good.
Can you describe the friction and its source? I'll do some rethinking and redesign.
Thank you for your interest!
No worries, this space has so much inefficiency that I'm sure you will eventually start chipping away at it.
> Can you describe the friction and its source? I'll do some rethinking and redesign.
Some software ecosystems already have their own build tools. Often these build systems are intertwined with IDE support. So care must be taken to ensure the IDE still functions as developers expect. If you ignore the functioning of the IDE, developers will balk at adopting the new build tool. There is some prior art for generating files that an IDE can understand:
https://buck.build/command/project.html
https://github.com/bazelbuild/tulsi
Interfacing existing build tools into a meta build tool can be difficult to do efficiently, but I’ve run out of time for talking about that tonight.
Rig will have multi-project builds. It also has a concept called "context stacks".
The idea would be that you would have a "compiler option" context stack, and you would push `-Wno-bad-warning`, then inside the context, build that library.
Then the library would add `-Werror`, but it would also have `-Wno-bad-warning`, so no problem.
But multi-project builds won't exist in the first release.
He also describes the dynamic discovery of header dependencies of Ninja as a hack. It is true that it is a special case to handle a specific type of dynamic dependencies. But the basic idea behind it (using the compiler's diagnostic as the source of truth) is sane and efficient. The alternative is to scan the sources using a custom parser, which is slower and more fragile.
More fragile, yes, but I dispute the "slower" aspect. waf does this very well. In C you don't need to parse the entire program structure, you just need to parse what the preprocessor does.
Also there's a difference between the complete set of header dependencies (notably #include statements within an #if or #ifdef) and a more sane but incomplete methodology for #include statements (don't do it within #if/#ifdef). Not sure whether this is a big enough difference to try to matter.
Yes, it is unfortunate that "make" syntax is a bit odd. But if you can't describe the hierarchical DAG for how your product comes together, that ought to be the problem to tackle rather than building a Turing complete language to try to describe how you think maybe your product might come together.
Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled.
Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache...
I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result.
Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz...
This right here. When working with large enterprise clients I'm seeing insanely complex build chains.
Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
Essentially you have a DSL. A DSL can be great if you use it frequently enough to know the insurance and outs. Like SQL.
But a build DSL is something you muck with like ... Once every six months? And most build files I one off rather than write from scratch. I'm not going to o learn your DSL. I'm going to stack overflow it.
And build systems no matter how genius prescient the authors are, do weird things sometimes. Integrate with so e proprietary library? Have a required code generation step? Pull a dependency from a weird location? Auto annotate code?
Alas then you have to either be able to decompile the DSL to the base lang.in your head, or inject some "real code".
And ... By the standards of this article when dealing with the problem of building code (which, come on, evolves to a workflow engine eventually, and not a simple one) becomes meta.
The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.
It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects.
I don't believe anyone has missed Turing completeness.
Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
Given some starting known true values in the system (e.g. debug_build=true, platform_is_windows=true, build_curl_library=true) you solve to find the minimum set of other variables that need to be true.
Each variable can then have some effects tied to it (e.g. if build_curl_library is true, then make sure curl.c is compiled and curl/include is in the include path)
Being Horn Clauses, the solving is basically a flood fill - as soon as you know a variable is true, it is locked in as true. If there was negation, then you can have weird contradictions and reasoning about the system starts being really difficult.
The build system is then able to figure out a starting set of booleans which must be true (e.g. "I am writing the makefiles for the unit test project so it builds on linux in a debug build"). This translates into some starting assignment of boolean values to the boolean variables.
However, it is most likely, that this assignment will not satisfy every horn clause. For example: "if (debug build) then (include debug symbols = TRUE)" might be a clause, and so known that we have a debug build implies we need to set our boolean for including debug symbols to be true (we can break this out so we can then easily have things like optimized builds with debug symbols). Then once you know that you have debug symbols, we might have a clause like "if (debug symbols AND compiling with gcc AND writing a makefile) then (<magic side effect goes here>)"
Similar stuff happens for dependencies. If some code uses the maths library, then the place where the magic side effect says "add this file to the list of files being compiled" will also have a "link with maths library = TRUE" part to it, so then anyone who sets the boolean to get that file, will magically also be told that they need the maths library, and some other part of the system will be responsible for figuring out what that means (e.g. adding "-lm" to some makefiles)
The syntax for it is all simplified down so it doesn't really look like you are doing boolean arithmetic everywhere, but the semantics are mostly what I'm describing.
And the answer to 'how to build this' should be answerable by "run build_system". Any further details represents a failure of the build system (what you describe of having multiple build systems for each platform/IDE is especially hellish). Even a relatively small amount of code is likely to have details the matter beyond just 'compile and link'. Especially in embedded, where even if the codebase is all the code that's running on the system (no external libraries), it's far from trivial to actually get a working binary just by invoking the compiler.
I am very familiar with embedded stuff, and this whole system was built precisely because of that. Some people are developing in visual studio so give them a good visual studio project that works well, some are working with a proprietary garbage toolchains, so you need to let them write makefiles and have enough flexibility to let them encode how to do that in the build system (e.g. all the different quirks you get for figuring out header file dependencies). The policy I used was "every toolchains needs some lowest common denominator support" and then "once you have that, let every toolchains be as good as possible with equivalences given abstractions". It did appear to work.
I think you may have misinterpreted me on the multiple build system thing. There is one build system and it produces files that each ide/toolchains can work with. So it can make xcode, visual studio, makefile, etc. No one edits those by hand, just like cmake, gyp, etc
My go-to example is multi-language builds. Those get complex, and sometimes, multiple languages are necessary.
I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
That said, if Rig gives you nothing new, please don't even try it!
I'd rather you wait until you need it to try it; then you'll be more forgiving of the bugs. :)
Specifically, this idiom:
all:
for dir in $(MODULES); do \
(cd $$dir; ${MAKE} all); \
done
module1:
(cd $module1_dir; make)
module2: module1
(cd $module2_dir; make)
Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies).
The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
I think the fundamental issue boils down to aggregation. Normally, aggregation is a good thing, a variant of the divide and conquer technique which we were all taught in kindergarten you can never go wrong with. But in build systems aggregation leads to the loss of precision (rebuilding more than necessary) or accuracy (not rebuilding when necessary, rebuilding things in the wrong order), and usually both. This is the fundamental reason why "recursive make is considered harmful": we split one big graph that has the total, accurate view of all the dependencies into a number of aggregate sub-graphs. And the same happens with meta build systems where we do some part of the build (configuration, pre-build steps where we generate source code, etc) in the meta build system and the rest in the underlying build system, essentially splitting the graph into two.
P.S. And, yes, `build2` supports both dynamic prerequisites/dependencies[1] and dynamic targets[2].
Did I miss something or did the authors actually didn't explain what prevents meta build systems to implement any of those features.
Looking at the current state of cmake it seems to have everything mentioned here. Including dependencies and header parsing in c++
https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...
> Alas, it cannot; the list of stuff to build is passed from the meta build system to the build system, usually by fiat.
It can, actually. At least for make. Just use a wildcard in uour rules: https://www.gnu.org/software/make/manual/make.html#Wildcards
I highly recommend going through the make documentation at least once in your career. Per the lindy effect, as it has been around for 40 years, it has a decent chance of sticking around for another 40.
I mentioned suffix rules and pattern rules.
Unfortunately, meta build systems don't like to generate rules like that.
I have worked with them long enough ago I can't remember.
Do they require the build files to list all dependencies?
The point of dymanic dependencies is that Rig could run some code to figure out what the main function imports, transitively, and use that info to build the entire project.
All without listing dependencies in the build file.
A good API might look like this:
build_java_package("com.yzena.rig");
IDEs provide tooling to suggest matching dependencies to be added to the build file based on the names of unresolved imports. However, the much more common direction is to first decide which libraries (dependencies) you want to use and add those to your build file, which then allows the IDE to suggest imports when you reference some unresolved type in your source code. So the dependency declarations in the build file generally come first, before you start writing your corresponding code. (This can be likened to specifying coarse-grained includes in your project build file, from which fine-grained sub-includes are then selected in each source file.)
The conscious selection of dependencies is a good thing, IMO. Dependencies also don’t change frequently, so there isn’t much work to be saved by determining them automatically (except for transitive dependencies upon version bumps, which are determined automatically by the build system). Moreover, it’s not only the dependencies as such that are consciously selected, but also their respective version numbers. This prevents unexpected breaking changes.
It's clear to me that any build system that's NOT based on a declarative file format and strong project model is a nightmare waiting to happen. Each generation of maintainer will add it's own customizations until it becomes a super specific brittle undocumented jumbled mess that's nigh impossible to work with and that prevents refactoring of project code.
I'll have to do a bit more investigating into what this article means by "dynamic dependencies".
Perhap just one layer of meta isn't enough...
<hides under carpet>
Every IDE is different, unfortunately.
Rig will have to adapt to each IDE, but it will generate files with enough info to call into Rig, which will do the job.
Speaking of Bell Labs, there are some great ideas for a build tool in the Bell Labs "nmake" tool - I believe it was originally created to speed up ESS5 switch builds, which took 3 days before make. It has programmable header file scanning for example, a shell coprocess to avoid all the fork/exec of standard make, build state files, lots of other things I have forgotten.
You'd think that would be hard for even engineers to screw up, but here we are.