Recursive Make Considered Harmful (1997) [pdf]
aegis.sourceforge.net
aegis.sourceforge.net
OK, maybe not. But this here autotools cleanly handles all those crazy portability edge cases and gives you a nice structure to boot. It's the Right Thing.
Sorry about that. But check out scons. That's clearly the way things should have been done in the first place.
Late breaking news: supporters of "Ant" in the Java world were just deposed by something called "Gradle"! We'll have experts on shortly to explain why either of these were needed in the first place given that javac handled dependencies automatically in its very first version.
But yeah, maybe python wasn't as clean as we hoped. Check out cmake! Here's the new hotness for sure. Pay no attention to the really awful programming language it embeds, it does windows!
----
I mean, yeah, recursive make, specifically, isn't a great idea. And for large projects, make implementations have performance issues.
But make is a tool that does two things and does them simple and well: it tracks dependencies between files, and it lets you write simple command sequences in an obvious language to fulfill those dependencies.
And until the authors of all these other tools get their heads out of the clouds and stop trying to fix the OTHER problems with make while refusing to preserve these attributes, make isn't going anywhere.
People experienced with make might not notice, but Makefiles often become write-only code, especially to novices.
Not that the problems in the paper here are not specific to make. They describe deficiencies in a certain kind of mathematical model. It is possible (and common) to have the same problems in a non-make build system. Make is not special in this regard, just popular.
Gah, that's exactly the fallacy I mentioned above! That's not the requirement, as shown by decades of make's refusal to die. The requirement is that it work simply and easily for the simple tasks needed to get new projects built in their early days where their structure is changing rapidly.
Other stuff tends to force you into a world view, which is why we have separate build systems for Android apps, and Rust modules, and cross-platform C++ code, and Java projects, and Linux command line tools, and...
That said: I don't know what you're talking about here. Make does a perfectly general DAG via its routine and simple dependency syntax. Recursive make invocations do not, which is what the linked article about. But make works fine for arbitrary dependencies. Go build a kernel to see.
Nobody denies that Make can model all this. CMake and autoconf actually embrace that quality of make. The concern is that turning "depend on libfoobar.a" or "libfizzbuzz.a now requires FIZZBUZZ_STRING=std::string" into a coherent Makefile change is more often nontrivial, especially without breaking parallelism, correctness, or the ability to do incremental builds.
Build of a what of a which does what now? Why on earth do I care about that? (No, don't explain it to me. I know the answer. I just don't care.)
We're clearly talking past each other. You're still hung up over obscure minutiae that giant projects need to worry about, and you're convinced that if you just had a tool that handled all that minutiae and that if everyone used it that everything would be roses.
I'm saying:
1. Your minutiae isn't my minutiae. Your favorite tool sucks for, probably, most people. Tools designed to contorted sentences like the one I quoted tend to only work for one thing. That's how you get ant and cargo and cmake, not a better generic build system.
2. Minutiae isn't the issue. All big projects end up stuck in minutiae details and it's always a mess.
3. Tools designed to address minutiae details tend to be outrageous garbage when faced with simple problems.
4. Most build problems are simple at their root, especially in the early days when you are still figuring out what the build really wants to be doing.
5. Make is, was, and remains a better tool for people faced with #4, which is why it won't die.
#2 may be universally true, but Make does seem to get you to the tipping point much sooner.
I've found #5 only true if the solution to #4 consists entirely of one phony target, listing the commands to be run, one after another. There's nothing wrong with using Make like this, and it does have some advantages over a Bash script with set -v -x -e at the top, but this isn't a great advertisement for Make's usability.
(In fact, regarding "if you just had a tool that handled all that minutiae and that if everyone used it that everything would be roses" - yes, I have found that everything is roses when you do this ;) Building stuff that uses Autotools or CMake, horrid as they are, is almost always pretty straightforward! Stuff that comes with a "simple" Makefile is rather likely to require at least some work. Even simple missing dependency/wrong compiler version issues result in much more cryptic error messages.)
You need topological sorting for that. And you need an accurate and detailed model of your build for that. You only need three or four moving parts in your project for that to become hard to maintain manually.
Most development automation problems are not simple. Especially if you want to be portable and readable. It seems really easy when you start but then you need to do debug builds, incorporate sanitizers, generate a little protobuf, link against libclang, run a linter, respect pkg-config, support install ands, or any of a number of other things.
These aren't especially academic or esoteric use cases. They're reasonable things to do in a C or C++ project.
Last I checked, make only handled dependencies with multiple inputs and a single output.
If your build step's command produced more than one output and you wanted to model that in your DAG you had to resort to some questionable incantations that never really worked perfectly (or rely on non portable constructs).
secondly, working around this was never very hard, albeit ugly: pretend your rule builds just one of the targets and have the other products depend on that one and simply touch themselves up to its timestamp.
https://www.gnu.org/software/automake/manual/html_node/Multi...
Definitely not handled "simple and well" as posited by the OP.
Some rather obscure makes that you and I will never get to use have had the feature.
GNU Make has, for a long time, supported multiple outputs in pattern rules. Pattern rules have the limitation that the files involved have to be organized around a common stem. If you can make your situation fit this restriction, then you can effectively have multi-target rules by way of pattern instantiation, and this will work with even old versions of GNU Make.
Some two months ago I added the feature to GNU Make to express direct rules with multiple outputs.
http://git.savannah.gnu.org/cgit/make.git/commit/?id=8c888d9...
So bleeding edge GNU Make lets you do:
y.tab.c y.tab.h y.output &: parser.y ...
x y z : w
1. Rule with multiple outputs are problematic. Stamp-files work as a hack, but the build ends up harder to maintain as developers need to figure out which stamp controls what files
2. There's no good solution to handle changes in the makefile. Particularly, if a make target disappears, there's no way to delete the output file. If another build step later on grabs the supposed-to-no-longer-exist file via a glob pattern, build failures or run-time fun can result. Workaround: attempt a clean rebuild whenever the build does something weird. This is especially problematic when switching between development branches.
3. Recursive makefiles suck because you can't have fine-grained dependencies between parts of the sub-projects. Non-recursive makefiles suck because it's just textual includes: there's no scoping, so name collisions between different sub-projects cause trouble. And all paths have to be relative to the top-level makefile, so there's no way to write a makefile that can be included in arbitrary other projects.
4. Some features have weird semantics that makes them unusable in practice; e.g. the "target: export A=B" syntax looks like it's setting an envvar locally for a single target, but actually also sets it for other targets "invoked by" the target. Of course, which dependencies of "target" are "invoked by" it ends up being non-deterministic in a parallel build.
Fun fact: even "target: private export A=B" doesn't work reliably; the "private" modifier in GNU make only fixes this problem for make variables, but not for environment variables.
So is there a build tool that fixes these problems, instead of just replacing the parts of make that actually work well?
I created something that tries to improve some aspects of Make. In particular it allows setting multiple outputs and use checksum instead of timestamp for build caching.
Don't expect a perfect tool, though. This started as a toy project, but evolved with time.
Break your make build up by recursively including make files instead of recursively invoking them.
Android suffered from this a lot in the early days. I always wondered whether the initial Android team had read this paper and got carried away...
but parsing, once all the text is in memory? peanuts.
Android's build files currently add up to 1.5gb of text(!!!), which even with a fast parser take some time to load (~10-20s).
It's discussed further in this thread (in which among other things I suggest that maybe something is going wrong when you have 1.5gb of text):
https://groups.google.com/d/topic/ninja-build/htlu9akobq0/di...
[disclaimer: I am the author of Ninja]
Personally, I switched to WAF a loong while ago and have never looked back. Perhaps it is not the best build system there is, but it is leagues ahead of autotools/make.
If this comment had a title it could be “Considered Harmful articles considered harmful” or “Bob is mad at considered harmful articles”. Someone else could come along and write “Lack of considered harmful articles considered harmful” with equal merit I’m sure. ;)
I’ve made a point the last few years to put at least one goto in all my code just to mess with the junior devs, though in practice the junior devs aren’t familiar with the original material so its more the mid level devs that will explode into green flame or curl into a ball when they find my goto. Senior devs couldn’t care less if they find a goto.
For other languages, once I found rake, I never looked back. It's so powerful, and unlike make, you don't have to drop down to other languages to do things.
A group/language/team decides "that's it, I hate build tool X, we're going to solve this problem", and they go build their thing, but they care about different details and make some things easy, handle some things automatically, and leave other things as "just don't care" with the resulting tool ends being just like all the previous ones.
I'd be willing to wager lots of money that in future years there will be many more build systems, and they too will perfectly match their predecessors.
http://boost.2283326.n4.nabble.com/CMake-Announcement-from-B...
It appears further discussions are underway, but I don't know what that means in terms of how quickly they will switch. https://groups.google.com/forum/#!topic/boost-steering/5ifzu...
In terms of bjam vs. cmake... I get why the boost library has needed a custom build tool chain (if you're curious just look at the complexity in headers in trying to work around various template bugs in different compiler versions, or the many many many ways to request naming of the generated libraries). I should have been less snarky in my first post. If boost does migrate to cmake I predict it will be quite a gnarly set of cmake logic.