Comment by ajross | Hacker News Reader

Comment by ajross | Hacker News Reader

Yup. But imake is here now and we can all rest easy.

OK, maybe not. But this here autotools cleanly handles all those crazy portability edge cases and gives you a nice structure to boot. It's the Right Thing.

Sorry about that. But check out scons. That's clearly the way things should have been done in the first place.

Late breaking news: supporters of "Ant" in the Java world were just deposed by something called "Gradle"! We'll have experts on shortly to explain why either of these were needed in the first place given that javac handled dependencies automatically in its very first version.

But yeah, maybe python wasn't as clean as we hoped. Check out cmake! Here's the new hotness for sure. Pay no attention to the really awful programming language it embeds, it does windows!

----

I mean, yeah, recursive make, specifically, isn't a great idea. And for large projects, make implementations have performance issues.

But make is a tool that does two things and does them simple and well: it tracks dependencies between files, and it lets you write simple command sequences in an obvious language to fulfill those dependencies.

And until the authors of all these other tools get their heads out of the clouds and stop trying to fix the OTHER problems with make while refusing to preserve these attributes, make isn't going anywhere.

What the paper is actually saying (been a while since I read it but this was my takeaway a long time ago) is that make works well, just don't call it recursively. For example if you use inclusion to add additional targets rather than re-invoking make recursively to build them, then make can do its job well, with an accurate view of dependencies, without a bunch of the pitfalls.

Agreed. Problems with make:

1. Rule with multiple outputs are problematic. Stamp-files work as a hack, but the build ends up harder to maintain as developers need to figure out which stamp controls what files

2. There's no good solution to handle changes in the makefile. Particularly, if a make target disappears, there's no way to delete the output file. If another build step later on grabs the supposed-to-no-longer-exist file via a glob pattern, build failures or run-time fun can result. Workaround: attempt a clean rebuild whenever the build does something weird. This is especially problematic when switching between development branches.

3. Recursive makefiles suck because you can't have fine-grained dependencies between parts of the sub-projects. Non-recursive makefiles suck because it's just textual includes: there's no scoping, so name collisions between different sub-projects cause trouble. And all paths have to be relative to the top-level makefile, so there's no way to write a makefile that can be included in arbitrary other projects.

4. Some features have weird semantics that makes them unusable in practice; e.g. the "target: export A=B" syntax looks like it's setting an envvar locally for a single target, but actually also sets it for other targets "invoked by" the target. Of course, which dependencies of "target" are "invoked by" it ends up being non-deterministic in a parallel build.

Fun fact: even "target: private export A=B" doesn't work reliably; the "private" modifier in GNU make only fixes this problem for make variables, but not for environment variables.

So is there a build tool that fixes these problems, instead of just replacing the parts of make that actually work well?

> So is there a build tool that fixes these problems, instead of just replacing the parts of make that actually work well?

I created something that tries to improve some aspects of Make. In particular it allows setting multiple outputs and use checksum instead of timestamp for build caching.

Don't expect a perfect tool, though. This started as a toy project, but evolved with time.

https://taskfile.dev/

> ...it lets you write simple command sequences...

People experienced with make might not notice, but Makefiles often become write-only code, especially to novices.

So does everything else. Find me a big project with a build automation system that isn't a jury-rigged, inscrutable mess. The problem isn't the tools.

If it's all a mess, why not pick a build system that lets you compose a dependency DAG correctly?

Not that the problems in the paper here are not specific to make. They describe deficiencies in a certain kind of mathematical model. It is possible (and common) to have the same problems in a non-make build system. Make is not special in this regard, just popular.

> that lets you compose a dependency DAG correctly

Gah, that's exactly the fallacy I mentioned above! That's not the requirement, as shown by decades of make's refusal to die. The requirement is that it work simply and easily for the simple tasks needed to get new projects built in their early days where their structure is changing rapidly.

Other stuff tends to force you into a world view, which is why we have separate build systems for Android apps, and Rust modules, and cross-platform C++ code, and Java projects, and Linux command line tools, and...

That said: I don't know what you're talking about here. Make does a perfectly general DAG via its routine and simple dependency syntax. Recursive make invocations do not, which is what the linked article about. But make works fine for arbitrary dependencies. Go build a kernel to see.

The problem with having one monster make system is that code changes (adding a new dependency) affect build rules indirectly (the build of a test of a downstream library now needs to add a new include search path in a topologically correct way). Now, there are various DRY ways to model that sort of thing in make. But very quickly, the project is no longer an intuitive expression of allegedly simple make. It is a build framework written in make.

Nobody denies that Make can model all this. CMake and autoconf actually embrace that quality of make. The concern is that turning "depend on libfoobar.a" or "libfizzbuzz.a now requires FIZZBUZZ_STRING=std::string" into a coherent Makefile change is more often nontrivial, especially without breaking parallelism, correctness, or the ability to do incremental builds.

> the build of a test of a downstream library now needs to add a new include search path in a topologically correct way)

Build of a what of a which does what now? Why on earth do I care about that? (No, don't explain it to me. I know the answer. I just don't care.)

We're clearly talking past each other. You're still hung up over obscure minutiae that giant projects need to worry about, and you're convinced that if you just had a tool that handled all that minutiae and that if everyone used it that everything would be roses.

I'm saying:

1. Your minutiae isn't my minutiae. Your favorite tool sucks for, probably, most people. Tools designed to contorted sentences like the one I quoted tend to only work for one thing. That's how you get ant and cargo and cmake, not a better generic build system.

2. Minutiae isn't the issue. All big projects end up stuck in minutiae details and it's always a mess.

3. Tools designed to address minutiae details tend to be outrageous garbage when faced with simple problems.

4. Most build problems are simple at their root, especially in the early days when you are still figuring out what the build really wants to be doing.

5. Make is, was, and remains a better tool for people faced with #4, which is why it won't die.

I expect you must use a different Make from the one I do? Points #2 and #3 much more accurately characterise my experience with Make than my experience with other tools.

#2 may be universally true, but Make does seem to get you to the tipping point much sooner.

I've found #5 only true if the solution to #4 consists entirely of one phony target, listing the commands to be run, one after another. There's nothing wrong with using Make like this, and it does have some advantages over a Bash script with set -v -x -e at the top, but this isn't a great advertisement for Make's usability.

(In fact, regarding "if you just had a tool that handled all that minutiae and that if everyone used it that everything would be roses" - yes, I have found that everything is roses when you do this ;) Building stuff that uses Autotools or CMake, horrid as they are, is almost always pretty straightforward! Stuff that comes with a "simple" Makefile is rather likely to require at least some work. Even simple missing dependency/wrong compiler version issues result in much more cryptic error messages.)

Knowing correct compile commands for your project isn't minutiae. It's basically the point of a build system. It's not as trivial to get them right as you seem to appreciate.

You need topological sorting for that. And you need an accurate and detailed model of your build for that. You only need three or four moving parts in your project for that to become hard to maintain manually.

Most development automation problems are not simple. Especially if you want to be portable and readable. It seems really easy when you start but then you need to do debug builds, incorporate sanitizers, generate a little protobuf, link against libclang, run a linter, respect pkg-config, support install ands, or any of a number of other things.

These aren't especially academic or esoteric use cases. They're reasonable things to do in a C or C++ project.

Kobalt build tool uses a Kotlin DSL (typed language) and lets you establish dependencies between files in a straightforward way. It currently only supports java and kotlin jvm projects but it meets your criteria.

https://beust.com/kobalt/home/index.html

yes, but have you tried writing macros in the GNU Make extension language and reading them the next day? dude.

> But make is a tool that does two things and does them simple and well: it tracks dependencies between files, and [...]

Last I checked, make only handled dependencies with multiple inputs and a single output.

If your build step's command produced more than one output and you wanted to model that in your DAG you had to resort to some questionable incantations that never really worked perfectly (or rely on non portable constructs).

firstly, the upcoming version of GNU Make will indeed support multiple outputs in non-pattern rules (pattern rules always supported this, but pattern rules, uhh, have their own problems), using &: instead of :, resolving a bug from 2004. don't you just love the solid pace of those old GNU projects.

secondly, working around this was never very hard, albeit ugly: pretend your rule builds just one of the targets and have the other products depend on that one and simply touch themselves up to its timestamp.

I disagree with the characterization that this is "not very hard" (and my initial assessment was "questionable incantations" which I think is also accurate):

https://www.gnu.org/software/automake/manual/html_node/Multi...

Definitely not handled "simple and well" as posited by the OP.

the horrible contortions listed on the linked page are required for the pathological "some of the targets that are built together got deleted, but the primary one did not" case, which in my experience Does Not Happen (or when it does, it is a symptom of deeply suspect practices -- like shitting where you ea^W^W^W^Wkeeping build artifacts and sources in the same directory, or relying on manually deleting artifacts to force a rebuild instead of getting the bloody dependencies right in the first place, etc.)

(I guess what I'm trying to say here is that the 80% solution is not nearly as hard as that linked page makes it seem).

POSIX make doesn't specify any feature for multiple outputs.

Some rather obscure makes that you and I will never get to use have had the feature.

GNU Make has, for a long time, supported multiple outputs in pattern rules. Pattern rules have the limitation that the files involved have to be organized around a common stem. If you can make your situation fit this restriction, then you can effectively have multi-target rules by way of pattern instantiation, and this will work with even old versions of GNU Make.

Some two months ago I added the feature to GNU Make to express direct rules with multiple outputs.

http://git.savannah.gnu.org/cgit/make.git/commit/?id=8c888d9...

So bleeding edge GNU Make lets you do:

  y.tab.c y.tab.h y.output &: parser.y ...

It had to be done via a bit of new syntax because

  x y z : w

already has a meaning: it means that x, y and z independently depend on w, not as a group built together.

so true, made my day

Reply on news.ycombinator.com