Hacker News Reader

Comment by girzel | Hacker News Reader

I recently decided it was time to get a better understanding of how makefiles work, and after reading a few tutorials, ended up just reading the manual. It's long, but it's very, very well written (a good example of one of the Gnu projects biggest strengths), to the point where just starting at the top and reading gives an almost tutorial-like effect. Just read the manual!

View on HN

FWIW I also read the GNU Make manual, and based some code for automatic deps off a profoundly ugly example it had. Then later people on HN showed me a better/simpler way to do it.

https://news.ycombinator.com/item?id=15060149

https://www.gnu.org/software/make/manual/html_node/Automatic...

After reading the manual and writing 3 substantial Makefiles from scratch, I still think Make is ugly and, by modern standards, not very useful.

In my mind, the biggest problem is that it offers you virtually no help in writing correct parallel and incremental builds. I care about build speed because I want my collaborators to be productive, and those two properties are how you get fast builds. GNU Make makes it easy to write a makefile that works well for a clean serial build, but has bugs once you add -j or when your repository is in some intermediate state.

> In my mind, the biggest problem is that it offers you virtually no help in writing correct parallel and incremental builds.

That’s interesting, because in my mind, parallel and incremental builds are the main features of Make, the features that it is best at, and all other features are secondary.

It sounds like your problem is with the correctness part. Make gives you no tools to enforce that your build rules are actually correct. Very few build systems provide any help here. What you are looking for is hermeticity. Bazel does this by sandboxing execution of all the build rules, and only the specified rule inputs are included in the sandbox. I recommend Bazel.

Otherwise, it is up to you to get your build rules correct, and switching build systems won’t help in general (although they may help in specific cases).

I find it surprising that you talk about using Make for a clean serial build, because if you want a clean serial build, you might as well use a shell script. Make’s only real purposes are to give you tools for incremental and parallel builds. Nearly any other tool you replace Make with will either have you sacrifice incremental/parallel builds or will give you the same hermeticity problems you would encounter with Make. Replacing Make, the main paths I see are towards improved versions of the same thing (e.g. Ninja, tup), completely redesigned versions of the same thing (Ant, SCons), systems and languages which generate makefiles (e.g. autotools, CMake), and the new wave of build systems which provide hermeticity (Bazel, Buck, Pants, Please). This last group is a very recent addition.

Mind you, Make is old and not especially well-designed, and it has plenty of limitations, but it’s good enough at incremental/parallel builds that it has stuck around for so many decades. Make is good enough at what it provides that replacements like Ant, SCons, Tup, Ninja, etc. don’t seem like much of an improvement.

Yeah I think we're in agreement, except that Ninja is not a replacement for Make.

Make is both a high-level and a low-level build tool (i.e. logic/graph description vs. execution)

Ninja is only a low-level build tool -- it focuses on execution only, punting logic to a higher level, which I like.

See my comments here:

https://news.ycombinator.com/item?id=19057836

I used Bazel/Blaze for many years, and even contributed to it a long time ago, and I agree it has many nice properties (although I'm more interested in building open source projects with diverse dependencies, which it isn't great for AFAIK.)

Another potential advantage of a build generator (described in that comment) is having one mode to enforce correctness of the build description, and another mode to be fast. In other words, you could generate a Ninja file with a sandbox like Bazel uses. You could use a tree of symlinks or run in a chroot with a setuid helper.

> Yeah I think we're in agreement, except that Ninja is not a replacement for Make.

I would definitely disagree with this. Make's higher-level logic/graph description features are only incidental, as Ninja proved, you can remove those features and end up with a tool that is equally valuable, more or less. To abuse an analogy, if I'm shopping for notebooks, I don't need to buy a notebook and pen shrink-wrapped together. I already have plenty of pens.

You're right that Bazel isn't good for open-source projects with diverse dependencies, but I think this is a problem that can be solved by developing some more infrastructure for that and writing the appropriate Starlark code (to be called from your your WORKSPACE file and create the appropriate repositories somehow). That code just isn't around yet.

All I'm saying is that Make was designed to be hand-written and often is, and Ninja was designed to be generated, and almost always is. See the manual:

https://ninja-build.org/manual.html#_philosophical_overview

Concretely, if you look at how Android or buildroot used GNU make, you can't do that with Ninja alone -- you need another tool.

GNU make is Turing complete, and Ninja isn't. IMO the Ninja design is better (which isn't surprising since it's not an accretion over decades).

I'd just like to point out that Ant, the only alternative to make you list that I've used extensively, is essentially a scripting system.

As I recall, the guy who developed Make wrote it in a weekend.

And never expected it to be so successful, and realized that he couldn't fix some of its problems because people were already relying on it.

That may be true, but GNU Make wasn't built in a weekend.

Just like a crappy Lisp interpreter can be built in a weekend, but an ANSI Common Lisp or Racket won't be built in a weekend.

Its not actually all that hard to make well-parallelized makefiles, provided you follow a few basic rules.

- Each build step has a unique artifact.

- That artifact is visible to Make as a file in the filesystem.

- That artifact is named $@ in the rule's recipe.

- Every time the recipe is executed, $@ is updated on success.

- If the recipe fails, it must return nonzero to Make.

- All of the dependencies of the artifact are represented in the Makefile

For example, here is how the format checks are run in my current project for some C code. Its mission: To verify that those source files which are under the aegis of clang-format are correctly formatted. BUILD_DIRS is a list of directories containing source code. CFORMATTER is the name of the formatting program. Not everything is under clang-format control, so FORMATTED_SRCS is used to opt-in to it.

BUILD_DIRS_FORMAT = $(addprefix .format-check/,$(BUILD_DIRS))

$(BUILD_DIRS_FORMAT): mkdir -p $@

# There aught to be a better way to control the suffix without becoming a match-anything # rule...

.format-check/%.c: %.c | $(BUILD_DIRS_FORMAT) $(CFORMATTER) $< -style=file > $@

.format-check/%.h: %.h | $(BUILD_DIRS_FORMAT) $(CFORMATTER) $< -style=file > $@

# Record the fact that each format check passed by touching a uniquely-named file. # note the call to `false` on error, since `echo` always succeeds.

.format-check/%.diffed: .format-check/% @(diff -u --color=always -- $* $< && touch $@) || \ (echo Formatting errors exist in $* ; false)

check-formatting: $(addsuffix .diffed,$(addprefix .format-check/, $(FORMATTED_SRCS)))

It's artifacts are:

- A formatted source file for each repository source file

- An empty file in the filesystem for each formatted source file that is identical to the repository source file.

- A tree of directories for the above.

Each format check is run exactly once, and only when source files change. If anything fails, then `make` returns nonzero and the build fails. Its also fully parallelized, since there aren't any neck-down points in the dependency graph. Every one of our pre-commit checks are structured this way. Build verification is as parallel as possible for fresh builds. Engineers can resolve and verify their problems quickly and incrementally when they fail.

You forgot the most important and most difficult part. Ensuring transitive dependencies really make their way into the makefile without huge manual effort. For C-like languages just adding an additional #include in a header file will break most makefiles. To solve it you need to generate makefiles using -MMD flags which are included from your main makefile and this is not very obvious.

As make was mainly intended to build C projects, not getting these batteries included is why i think parent, and many others, consider makefiles needlessly complex.

That's a nice set of rules, although I don't think they're all easy to follow or verify that 1000 lines of Make is following.

I considered 3 main use cases, and I wrote Makefiles from scratch for all of them. Make works to an extent for each case, but I still have problems.

1. Building mixed Python/C app bundles [1]

2. Building my website [2]. Notably people actually do complain about Jekyll build speed, to the point where they will use a different system like Hugo. So incremental/parallel builds are really useful in this domain!

3. Doing analytics on web log files (e.g. time series from .gz files)

One thing I didn't mention is that they all involve some sort of build parameterization or "metaprogramming". That requirement interacts with the problem of parallel and incremental builds.

For example, for #1, there is logic shared between different bundles. Pattern rules aren't really expressive enough, especially when you have two dimensions. Like (app1, app2, ...) x (debug, release, ASAN, ...)

A pet peeve of mind is having to "make clean" between a debug and a release build, and nearly all usages of Make have that problem, e.g. Python and bash's build system. You could say they are violating your rules because each artifact doesn't have a unique name on the file system (i.e. debug and release versions of the same object file.)

Likewise, Make isn't exactly flexible about how the blog directory structure is laid out. I hit the multiple outputs progblem -- I have Jekyll-style metadata at the front of each post (title, date, tags), so each .md file is split into 2 files. The index.html file depends on all the metadata, but not the data.

All of them have dynamic dependencies too:

1. I generate dependencies using the Python interpreter

2. I add new blog posts without adding Make rules

3. I add new web log files without adding Make rules

Make does handle this to an extent, but there are definitely some latent bugs. I have fixed some of them, but without a good way of testing, I haven't been motivated to fix them all.

I wrote up some more problems in [3], but this is by no means exhaustive. I'm itching to replace all of these makefiles with something that generates Ninja. It's possible I'll hit some unexpected problems, but we'll see.

My usage is maybe a bit out of the ordinary, but I don't see any reason why a single tool shouldn't handle all of these use cases.

[1] Rewriting Python's Build System From Scatch http://www.oilshell.org/blog/2017/05/05.html

[2] http://www.oilshell.org/site.html

[3] Build System Observations http://www.oilshell.org/blog/2017/05/31.html

> A pet peeve of mind is having to "make clean" between a debug and a release build, and nearly all usages of Make have that problem, e.g. Python and bash's build system. You could say they are violating your rules because each artifact doesn't have a unique name on the file system (i.e. debug and release versions of the same object file.)

There are at least two ways that this problem can be addressed. One is to support out-of-tree builds, one side directory per configuration. Builds based on the autotools do this by default.

The other is to use a separate build directory per configuration within the build. My current project uses local in-tree directories named .host-release, .host-debug (including asan), .host-tsan, .cross-release, and .cross-debug. All of them are built in parallel with a single invocation of Make, and I use target-scoped variables to control the various toolchain options.

The engineer's incremental work factor to add another build configuration isn't quite constant time, since each top-level target needs to opt into each build configuration that is relevant for that target.

> I hit the multiple outputs problem

I wouldn't really classify that as a problem in GNU Make, as long as you can specify the rule as a pattern rule.

I hear you on the testing problem. Make certainly behaves as if the Makefile's correctness isn't decidable. Even if you levied the requirement that a Makefile's decidability was predicated on the recipes being well-behaved, I'm not sure that the correctness is decidable.

> offers you virtually no help in writing correct parallel and incremental builds.

The key problem there that Make has no idea about the semantics of the shell code that appears in the build recipes. It has no idea how two build recipes interact with each other through side effects on objects that are not listed as targets or prerequisites.

I think ClearCase's clearmake (GNU-compatible) actually intercepts the file system calls (because the build happens on a ClearCase mounted VOB). So it is able to infer the real inputs and outputs of a build recipe at run-time. For instance, it would know that "yacc foogrammar.y" produced a "y.tab.h" even if the rules make no mention of this. So in principle it's possible to know that one rule is consuming "y.tab.h" (opens it for reading), that is produced by another rule (that wrote it), without there being any dependency tied to this data flow.

The interception could be done by injecting shared lib wrappers, I suppose. Our friend LD_PRELOAD and all that.

Of course, if we fix the parallel build with proper dependencies, a fixed incremental build also pops out of that.

There have been systems that take this approach such as fabricate.py (https://github.com/brushtechnology/fabricate) and tup (http://gittup.org/tup/).

GNU Make makes it easy to write a makefile that works well for a clean serial build, but has bugs once you add -j or when your repository is in some intermediate state.

So true. I wrote about this and solutions here: https://www.cmcrossroads.com/article/pitfalls-and-benefits-g...

Thanks, I bought your GNU Make book a couple years ago and read pretty much the whole thing! Along with reading the GNU Make manual a few years before that, I made an attempt to "give Make a fair shake".

I liked it, but it's honestly weird to me that a book published in 2015 can improve on the state of the understanding of a tool with 40 years of heritage :) I also get this "groundhog day" effect after about 10 years of seeing new GNU make tutorials / best practices on Hacker News every couple months. Everybody who learns Make has to go through this same thing.

-----

I read that you reimplemented GNU Make for Electric Cloud and I was impressed by that :) My project Oil [1] is a similar sort of project. It can run thousands of lines of unmodified shell/bash scripts found "in the wild". I got big distro scripts working last year, and I got thousands of lines of interactive completion scripts working recently, which I need to blog about.

For awhile I thought it would be nice to replace GNU make too, although (1) that's a lot of effort, and following Oil's strategy isn't possible since Makefiles can't be statically parsed and (2) I think it's happening anyway.

Major build systems are now split up into high-level and low-level parts, i.e. autoconf generating Makefiles, CMake generating Makefiles/ninja files. Android used to be 250K lines of pure GNU make (including GMSL), but now it's a high level Blueprint DSL generating Ninja too.

There aren't many pure Make projects anymore, at least in open source code. The exception I can think of are embedded ones like buildroot.

I like how Ninja focuses on build execution only, punting logic to a higher level (CMake, gyp, Blueprint). So my pet theory is that you can replace Make with a DSL that generates

1) A ninja file for fast, parallel, incremental developer builds

2) A shell script for portable builds for distro packagers/end users. This is just a clean serial build, so it can be a shell script rather than a Makefile.

I plan to test this theory by throwing out the modest 700 lines of Make I've written from scratch and replacing it with a Ninja/shell generator :)

I've debugged and read enough Make to be able to identify and fix most problems. But it still feels like like whack-a-mole to me. You can fix one problem and introduce another, since there's no real way to test for correctness (and efficiency). Problems can be reintroduced by seemingly innocuous changes.

[1] http://www.oilshell.org/blog/2018/01/28.html

  > 1) A ninja file for fast, parallel, incremental developer
  > builds
  >
  > 2) A shell script for portable builds for distro
  > packagers/end users. This is just a clean serial build, so
  > it can be a shell script rather than a Makefile.

If the developer isn't regularly using the same build that downstream users are, the build for downstream will be perpetually broken.

It's no different than CMake generating Makefiles or Visual Studio projects, or autoconf generating Makefiles, which is the state of the art in open source.

I'm not saying you write them by hand -- you generate them from the same build description, and the generator can preserve some invariants. It should basically do a topological sort ahead of time rather than at runtime.

The more likely source of breakage is that the user's environment is different, i.e. they don't have a particularly library installed). So even if you choose pure GNU make, you still have that source of breakage, and you generally should test for it. I test my shell in a chroot with different versions of libc and without GNU readline. I need that even though I'm using pure GNU make at the moment.

Yes. It is highly recommended that you read that book (buy it to support the FSF). If you want more I wrote a book on GNU Make (https://nostarch.com/gnumake) which takes things further than the GNU Make manual. A large amount of the content of the book came from a sequence of blog posts on GNU Make by me: https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-art...

Your book is awesome, thanks for writing it! I converted our whole build at $DAYJOB from recursive to non-recursive, and your book helped immensely. It was a lot of work, but in the end it was worth it.

That's very kind. Weirdly, I have never done a conference presentation about GNU make. If I did what would people want to hear about?

I think the main point nowadays is how to write Makefiles for highly parallel builds. It's surely the thing that gave me the most headaches. In the end such a talk will probably mostly be about how to avoid calling Make recursively. Another talk I'd like to hear, which would be more meta: does it still make sense to bother with Make at all, aside from maintaining legacy systems? Are there features or paradigms which still make it worthwhile, compared to other popular build systems?

I don't think the FSF is currently selling physical copies of the Make manual.

Ah. I guess I am old and like books. Give them some money instead: https://my.fsf.org/donate/

Can you humor a dumb question and tell me how you got the manual in your terminal? I'm on Debian, and when I `man make`, I just get a typical GNU CLI util manpage that's five paragraphs of intro and a line-by-line explanation of each command line option. Same for `info make`.

Have you installed the make-info (https://packages.debian.org/jessie/make-doc) package? It adds info documentation as well as pdf and html formats.

Brilliant, didn't even realize it was a separate package. Thanks!

I really like the GNU Make documentation, as well.

Links to the docs in various form:

https://www.gnu.org/software/make/manual/

What if I want to use makefiles on a mac? Make of macs is not gnu make.

/edit: bugger, after a make --version on a mac, i've got:

Back to the begining, why is this bloody thing not working...

FYI, homebrew installs version 4.2.1. Invoke it as gmake.

Reply on news.ycombinator.com