“The GNU Make Book”: probably more than you ever wanted to know about make | Hacker News Reader

“The GNU Make Book”: probably more than you ever wanted to know about make

blog.jgc.org

214 pointsjgrahamc66 comments

Implicit makefile rules can be quite helpful in quickly building a few binaries or small projects.

Take a look at this C++ example: for a file mybin this creates mybin.o and then links it, spitting out a mybin executable:

    env CXXFLAGS="-std=c++14 -Wall -Wextra -pedantic" LDLIBS="-lstdc++" make mybin

To see what's going on, and what kind of file types are supported:

    make --print-data-base | egrep 'COMPILE.cc|LINK.cc'

This not only shows you what is going on, but also what kind of environment variables are involved and can therefore be customized.

For small projects I would create a config.mk (e.g. with CXXFLAGS, CXX, ...) and provide a small Makefile, something along the lines of:

    include config.mk

    all: mybin

    mybin: mybin.o

    watch:
        while ! inotifywait -e modify *.cc; do make; done

    clean:
        $(RM) *.o mybin

    .PHONY: all watch clean

This allows you to easily 1/ add dependencies for binaries and 2/ speed up the development proccess, using the make watch target (note: this does not include header dependencies).

Disclaimer: Makefiles seem to be wonderful build-systems (not dependency managers!) for small projects; but as soon as you find yourself searching for hacks to build into your little Makefile, ditch it and switch to something serious. That's at least my experience.

> something serious

What do you recommend? Autotools, bespoke shell scripts, or something else entirely?

For C++, unfortunately CMake is the best option there is it seems. For LaTeX there is latexmk. Clojure has Leiningen.

You see, it heavily depends on your preferred environment.

FWIW when I tried out qmake for a toy c++ project, it's the first make tool for c++ that actually just worked[1] for a simple command-line app. CMake works too, but not without fighting the horrible syntax of CMake first. And make... well it works, but at what price to your soul? ;-)

[1] The documentation is actually a little dense, as qmake is a tool capable of building "real projects", but I found that a simple qmake -project && qmake && make (I think, at least you need a project file, and the let qmake make a makefile, and then use that...).

http://doc.qt.io/qt-5/qmake-running.html

premake is another tool that just works for C++, but it's not as flexible as some of the other build systems and so I don't use it.

At last, I find someone who uses qmake on HN. I've been using qmake for 4 years now. At first I started experimenting with it for a small project. The learning curve was quick and eventually fell in the comfort zone it created.

Now I use it for large scale C++ projects and I've never looked back for other build tools, not even CMake.

Surprisingly, I never understood why KDE project that uses the complete Qt ecosystem doesn't use qmake but rather settled to CMake.

Does qmake deal with finding the dependencies and automatically adding the include paths, libraries, etc. for them?

As far as I know, you need to stick the compiler and linker flags manually into qmake, which basically means that it works on your machine, but may break on anyone else's system. This is what a lot of the complexity inside cmake is addressing - differences in build environments.

I think the answer is: "It depends.". From some searching, it looks like cmake has better cross-platform support, and is (unsurprisingly) more powerful when you need to do something a little gnarly. That said, it appears (never had need to test) that qmake works fine with pkg-config:

http://doc.qt.io/qt-5/qmake-project-files.html

In my experience (mostly from compiling from upstream source, when something isn't available in Debian and/or backporting for personal use) there are different kind of libraries, some behave better than others (eg: easy to install under $HOME/opt with xstow -- as I prefer to /usr/local, as the latter needs root and/or rw-privileges on /usr/local). I've yet to find any pattern for when things just work, and when things don't (the real reason typically being some hard-coded paths or other nastiness -- I just mean some obscure projects work fine, some big ones fail miserably). So I guess YMMV -- but for now, qmake is the tool for building simple c++ I've found that is simple, for simple projects. I also use CMake (preferably with ninja) -- but I don't really like the CMake "language". Maybe what we need is a CMake-generator? Then we can generate CMake-files that generate ninja files that build our code! ;-)

I've had good results with qmake, too, but I don't know how well it works for large projects. I don't think any of the codebases I've used it on was > 100,000 LoC.

It will depends on you platform and technology. I can only recommend you to _not_ be fancy and stick with the conventions or else you could have a lot of trouble to interact with dependencies. By dependency I mean anything your program would need to run: from system libraries, headers, local libraries linked either statically or dynamically, or even syscall. A lot of things can go wrong, our whole software cathedrals are build upon years of conventions and patches. Anything can break. Don't even try too look what `ld(1)` is doing.

Autotools has never be a great tool, only a necessary evil for when you need to package cross-Unixes software. I heard that CPack (http://www.cmake.org/Wiki/CMake:Packaging_With_CPack) do the job while being exactly as good as CMake (which can be a compliment... or not. YMMV).

So, if you use Rust, use Cargo, with Java use Maven, with C# use MSBuild, with ruby use Rake, etc. And C/C++ ? Well, for once if you can chose you're lucky. You can try CMake or premake.

The only thing you must do is get away of most of the Google's build projects: Gyp, Lunch, Ninja, etc. I have horrible experiences with those. Even worse than Rubygem's native compilation, which is quite a performance.

What's wrong with ninja? It's CMake's best output format as far as I know. A lot more enjoyable to use than, say, msbuild.

For C++, I'm a long-held adherent of SCons.

You can make some horrendous messes but it's nice to have Python around instead of having to learn another intermediate language as in CMake's case.

I'll make my reply short:

> adherent of SCons.

o_O

> You can make some horrendous messes

:nod:

In fairness, you can do that in every build system. All build systems are awful. Pick the one you know best.

This is a good question. After too many years experience of fighting with build systems, I have come up with one overarching requirement - the build system needs to be debuggable. I need to be able to easily identify why a particular file was built. I need to be able to easily recover the command line that builds a generated file, as in if I know that I have a source file called foo.c, I need to be able to easily obtain the command line that compiles foo.c to foo.o, so that I can see the options that were used. I need to be able to easily see how variables are constructed. Something like being able to put a watch on them, so I can find the lines of build files that modify them.

You need these things because you know what? One day someone else is going to want to modify your build system, to cross-compile for another target device, or to build a library instead of an executable, or to add a configuration option, or whatever. Or they are going to find that they need to optimise the build because it's too slow. And without these tools, those tasks are ridiculously complicated.

And lastly, it, would be wonderful if I could take an autotools or pure-makefile based project, and convert it to this system with a simple import statement - the resulting project build file wouldn't need to be particularly beautiful, because, with all those other tools that I listed above, I could clean things up by hand after the import.

The bad news is that as far as I am aware, no such tool exists. I assume it must be a hard problem to solve, because the pain that build systems cause is a real and ongoing concern for millions of developers around the world.

If you're the sort of person that writes these types of system, I would be your customer. Make the basic build system - where you can create and build projects - free, and make the debugging tools something that you have to pay for. I for one would be your customer.

All of which leads me to the bad news: I've never yet encountered a system that does these things, which is why I needed to include a plea to someone to make one.

I like your watch target. I implemented something similar [0] with watch (bash builtin or GNU watch) & time.

   start_ci:
   	watch time -p make clean all & echo $$! > tmp.ci.pid

   stop_ci:
   	kill -9 `cat tmp.ci.pid`

Then you can start continuous integration by doing

   make start_ci

Then you can stop continuous integration by doing

   make stop_ci

Obviously, this is inspired by rc.d scripts.

[0] https://github.com/lpsantil/rt0/blob/master/Makefile#L117

aside: Please no `backticks` $(cat tmp.ci.pid) is standard and never worse than backticks. Indifferent in your example but frequently it's vastly safer. backticks need to be deprecated.

whilst this is good advice for shell scripting, are you sure it applies as worded to the snippet you're replying to, given how Make uses $() itself for variable referencing?[1]

$$() might work, but untested.

[1] https://www.gnu.org/software/make/manual/html_node/Reference...

yes, absolutely $$() works in a makefile. Backticks just need to die.

yea, at least with GNU Make it is probably best to use $(shell cat tmp.ci.pid)

Haha. Good catch, I actually use $(shell cat tmp.ci.pid) to capture the PID in a var $(TMPCI) if you look at the actual Makefile I linked.

+1 for teaching me about the inotifywait command.

I've had good luck with self-writing makefiles, where I write macros to generate the rules. Here's an example:

http://sourceforge.net/p/wordgrinder/code/ci/default/tree/Ma...

While it looks crazy, it has the advantage that it avoids pattern rules completely, which means that everything's explicit and we get to avoid some of make's more annoying misfeatures. It makes the makefiles vastly easier to reason about, as well as allowing things like building the same binary multiple times with different flags, which is really hard with traditional makefiles. The downside is that it looks crazy (and GNU make macros are not what you'd call well-designed, either).

As for switching to a different build system... like what? Every other build system I've seen is horrible in other ways, and make is at least ubiquitous. Every time I need to install some software which uses a non-make build system I get this horrible sinking feeling as I have to go and install the thing I need to install the thing.

...

Incidentally, the craziest thing I ever did in make was this:

http://cowlark.com/2013-10-19-insane-make/index.html

I wrote a compiler for a statically-typed language which targeted GNU make macros as a backend. Surprisingly, it actually worked, although I never got round to finishing the arbitrary-precision maths library needed by the runtime system...

The author is a master. I first encountered his byline when trying to debug a Makefile (that I had written!), and came across his article in DDJ (http://www.drdobbs.com/tools/debugging-makefiles/197003338). It was invaluable, because I had been stuck between using make -d and @echo "building foo now", and I hadn't wanted to admit I was actually debugging and so I needed some specialized tools. I'm sure the book is excellent.

A general comment on debugging GNU Make: there's a fork (http://bashdb.sourceforge.net/remake/) that adds interactive debugging facilities to GNU Make. Since this is a fork and not a reimplementation, it's 100% compatible. Much nicer than "make -d" generally.

Yeah, but it's based on an old version of Make; we're on to 4.1 now, it's still based on 3.82.

True. You're unlikely to be using anything that's in 4.1 but not in 3.82, however. I wish the remake stuff would get merged into the main tree, but there're probably political issues there, or something.

https://github.com/rocky/remake the sources there claim to be from version 4.1

blushes

If anyone (the author or anyone else!) wants to demonstrate their make-fu, here is a very subtle issue I ran into with make that I never did get to the bottom of.

http://stackoverflow.com/questions/25589586/why-does-patsubs...

Someone pretty knowledgable chimed in, but we were both stumped. In the end I found a workaround, but never did get to the bottom of this peculiar behavior.

The confirmed answer looks correct to me. I think your last comment on it is also correct.

The example from the GNU Make manual for SECONDEXPANSION uses patsubst but is not a pattern rule.

You tried to use it with a pattern rule, but then the pattern rule interfered with the % in the patsubst, replacing it before the second expansion, which is when the patsubst ran.

All that said... you could probably re-architect your Makefile a bit to make it simpler. I bit more verbosity might be worth it to avoid SECONDEXPANSION tricks.

    > EDIT: I had a theory that the % characters inside the patsubst
    > were getting evaluated early, using the stem match (foo), so
    > that the patsubst itself was looking something like this:
    >
    >    $(patsubst foo.c,foo.o,bar.c baz.c)

That is what happened. The answer on SO explains how to work around it.

    > To test this theory, I added a file called foo.c to foo_SRCS:
    ...
    > That resulted in something even weirder:
    >     make: *** No rule to make target `foo.a', needed by `all'.  Stop.

Which resulted in Make seeing the line as

    %.a: %.o bar.c baz.c

Now, Make will only choose to use a pattern rule if it knows how to make all of the prerequisites. If Make doesn't know how to make foo.o (which it wouldn't if you don't have an actual file named foo.c), then it won't even consider the rule as a possibility to make foo.a.

If it prevents the invention of the next Maven, Buildout or Grunt, I'm sold.

Also see O'Reilly freely available GNU Make book: http://www.oreilly.com/openbook/make3/book/index.csp

One thing I've never figured out a good solution for in make, is multiple-pattern rules. Say I have

  LANGS=nob sme sma smj
  POS=V N A

and want to run something on all combinations of these, like

  ./something --pos=V --f1=nob.f --f2=sme.f nob.txt sme.txt > nob-sme.V

– how can I make a pattern goal for that? I invariably end up with some redundancy, where e.g. only the "V" in the example above turns into my % (and $ * ), e.g.

  nob-sme.%: nob.f sme.f nob.txt sme.txt pos/%
          ./something --pos=$* --f1=nob.f --f2=sme.f nob.txt sme.txt > $@

Fortunately I haven't yet had to do this for anything too large to just copy-paste stuff, but it feels like something someone would have solved at some point, I've just never seen examples like that in any of the make-alternatives I've looked at.

You can't have more than 1 pattern variable but if they are all of the same form you can generate all these rules via $(eval); first define a function that generates one rule, then invoke it for all combinations.

Unfortunately this sort of thing is not very readable because make function parameters are numbered, not named.

  define genrule
  $(1)-$(2).$(3): $(1).f $(2).f $(1).txt $(2).txt pos/$(3)
    ./something --pos=$(3) --f1=$(1).f --f2=$(2).f $(1).txt $(2).txt > $$@
  
  endef

  $(foreach lang1,$(LANGS),\
     $(foreach lang2,$(filter-out $(lang1),$(LANGS)),\
         $(foreach pos,$(POS),\
             $(eval $(call genrule,$(lang1),$(lang2),$(pos))))))

Wow. I had no idea you could have that kind of meta-rules in make, nor am I sure I really want to :-) but that's still pretty neat.

I wonder what this book's take on the classic Peter Miller's paper "Recursive `make` considered harmful" [1] is.

[1]: http://aegis.sourceforge.net/auug97.pdf

The sample chapter might give some hints:

http://www.nostarch.com/download/GNU_Make_sample_ch4.pdf

Please note: Despite the title, this is not the official GNU Make manual, and is not associated with the authors of GNU Make or its manual. (Just in case anyone was confused.)

The author of the book is also the author of the more-confusingly-named "GNU Make Standard Library" :)

That said, I'd bet money that Paul Smith (the maintainer of GNU Make) would vouch for John Graham-Cumming's expertise.

I wonder why you would want to learn "GNU make" in contrast to POSIX make when one of the advantages of make is that you have it on all unix systems.

What computers don't have access to GNU make? I don't see why you would give up all those extra features in order to be a tiny bit more portable.

The BSDs don't use GNU make (OS X interestingly does). You can also install other build systems, the advantage with posix make is that you don't have to on unix systems.

Doesn't BSD make also have it's own features that are not portable?

GNU make isn't part of the base system for the BSDs, but I've found that it is often installed on BSD systems anyways (as gmake), due to much software having a build dependency by using Makefiles with GNUisms.

If your project is targeting a generic POSIX environment with build tools available and you want the portability, I can see why you might go with the lowest common denominator. For a lot of projects though, there are additional non-base dependencies (e.g. a python interpreter), and having a build dependency on GNU make is usually acceptable if you're using the extra features (e.g. % matching is a big convenience).

Conversely on a GNU system (like Debian GNU/Linux) you need to explicitly install BSD make... and on the *BSDs you have GNU in ports, and on windows you have it via MSYS/whatnot and on Solaris... etc. I suppose you don't have either on Android/CromeOS -- but the generally lack build tools anyway, as does iOS.

Reminds me of the days when we really needed to support old versions of AIX and HP-UX 9.x… So I guess in addition to not using GNU make, you'll have to be careful about your compiler supporting ANSI C, too.

C99 support is part of current versions of posix.

Which one is POSIX make?

This is also lowest common denominator computing. If you work on a system or environment that will provide certain set of features why would you limit what you can do or learn based on what is possible for everyone? Should we also be limiting our belongings to the space of other peoples living conditions? Should I ride a mountain bike in the city because my road bike doesn't like rough terrain?

If you only want to target that system then it's fine to use every capability it offers. If you want to target more systems you have to either work with the lowest common denominator or write several times the same operating system specific code for each OS.

GNU make is still make, which in my opinion has quite some warts. So for my view portability is the biggest reason why to use make. If you want to be portable and the features outlined in the posix standard aren't enough for you, you might as well use something else.

I'm confused which make is the POSIX one. I have bmake on my linux boxes to handle the BSDisms, because it has features that are useful and not POSIX. If you don't like gmake, I get it.

But let's not pretend there's a commonly used make that only observes POSIX, I think the ones I've seen {b,g,n}make, makeapp and are mostly POSIX but non quite.

There is no POSIX make, POSIX is just a standard. What I mean is that you should only use features outlined in said standard. This allows you to use your makefiles on all makes that are POSIX conform.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ma...

Many common fairly basic features are missing from POSIX make, like pattern rules and conditionals (if/ifeq/ifdef). Some were added just in the last year or two, like including possibly non-existing files.

see http://www.dwheeler.com/essays/make.html

GNU Make tends to be available on all systems, include OS X and the BSDs.

GNU make is everywhere I need.

If it isn't then porting GNU make is almost always the shorter path to getting something working.

POSIX make is so feature starved as to be nearly useless - it does not even have conditionals, and of course it lacks advanced features like $(eval) and target local variables.

Since you can't write a non-trivial makefile without conditionals, restricting yourself to POSIX make will lead you down the path of despair of using something like GNU automake.

OTOH, GNU make works even on Windows (with jobserver too!), so why bother with inferior alternative implementations.

I'm waiting for jgrahamc to post a 40% off No Starch Press coupon code for the loyal HN readers (in celebration of his newly-published book, that is :).

Btw, for people that are somehow stuck with Microsoft's make - "nmake" and its syntax, but lack threading (e.g. make's -j), you should try out Qt projects jom. It was mainly done so Qt and rest of the projects recompiles faster for Windows. It can even integrated with Xoreax's IncrediBuild:

http://wiki.qt.io/Jom

It's wonderful!

"having written a complete emulation of GNU make".

The conversation in the office on this is fascinating... writing a complete emulation based on how the GNU Make Manual says https://www.gnu.org/software/make/manual/ .

I'm curious how many HN readers use make to manage their frontend builds, similar to the process described here:

http://www.sitepoint.com/using-gnu-make-front-end-developmen...

I do mostly back-end stuff, but some of my projects use Make for minification/bundling during the deployment cycle. The good thing is that it's simple, easy to maintain and it works - and I am no GNU Make expert.

I just did a project with a Python backend and JS single-page-app frontend. My Makefile looks like this: https://gist.github.com/btubbs/c790111ff24357293ee2

I'm happy so far. I'm an expert in neither make nor Grunt, but I'd still rather write a Makefile than a Gruntfile.

Not to mention the proverbial simplicity of Unix utilities ...

I need one of these but for cmake =)

http://www.amazon.com/Mastering-CMake-Ken-Martin/dp/19309342...

I remember enjoying reading this book. If you're wanting/needing cmake information, find a copy of this book. Hopefully you enjoy it as much as I did.

Reply on news.ycombinator.com