Hacker News Reader

Comment by kccqzy | Hacker News Reader

No one wants to manually do dependency management in even a moderately sized project. I really haven't found an ideal way to have these -MM -MT flags integrated into Makefiles; I've tried having an awk script automatically modify the Makefile as the build is happening, but of course the updated dependencies will only work for later builds, so it's only good for updating the dependencies. Any other approaches HNers used and really liked?

View on HN

There exists DJB's redo approach [0], which i implemented [1], where dependencies and non-existence dependencies are only recorded after the build. A typical dofile is a shell script, so you do not need to learn another language. Targets also automatically depend on their own build rules (I have seen such a thing only in makefiles authored by DJB).

I wrote a blog post to show how to integrate dependency output for both dependency and non-existence dependency generation [2]. The game “Liberation Circuit” can be built with my redo implementation; you can output a dependency graph usable with Graphviz [4] using “redo-dot”.

There is only one other redo implementation that I would recommend, the one from Jonathan de Boyne Pollard [5], who rightly notices that compilers should output information about non-existence dependencies [6].

I would not recommend the redo implementation from Avery Pennarun [7], which is often referenced (and introduced me to the concept), mainly because it is not implemented well: It manages to be both larger and slower than my shell script implementation, yet the documentation says this about the sqlite dependency (classic case of premature optimization):

> I don't think we can reach the performance we want with dependency/build/lock information stored in plain text files

[0] http://cr.yp.to/redo.html

[1] http://news.dieweltistgarnichtso.net/bin/redo-sh.html

[2] http://news.dieweltistgarnichtso.net/posts/redo-gcc-automati...

[3] https://github.com/linleyh/liberation-circuit

[4] https://en.wikipedia.org/wiki/Graphviz

[5] http://jdebp.eu./Softwares/redo/

[6] http://jdebp.eu./FGA/introduction-to-redo.html#CompilerDefic...

[7] https://github.com/apenwarr/redo

Simply add:

  -include $(OBJS:%.o=%.d)

in your makefile (with -MMD in CFLAGS).

This is what I do too, and it seems perfect to me for adding set-and-forget dependency awareness to an ordinary Makefile. The first build with a new .c file will work. Its .d file won't exist, but the -include directive silently ignores this because of the - prefix, and it will be built anyway, because its corresponding .o doesn't exist either. Subsequent builds will use the dependency information in the .d.

Also consider adding -MP to CFLAGS to prevent errors if you delete a .h file.

Don't you need a pattern rule for %.d? GNU Make doesn't seem to come with this rule.

There is a profoundly ugly example here:

https://www.gnu.org/software/make/manual/html_node/Automatic...

    %.d: %.c
        @set -e; rm -f $@; \
         $(CC) -M $(CPPFLAGS) $< > $@.$$$$; \
         sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@; \
         rm -f $@.$$$$

I copied it into a running example here for anyone curious:

https://github.com/oilshell/blog-code/tree/master/dotd

If you have -MD or -MMD in your CFLAGS, GCC (and Clang) will generate the .d files during compilation without you having to add anything else to the Makefile, and without requiring ugly sed magic.

I just tried this, and it seems wrong on two counts:

1) The sed magic is required for adding the dependencies of the .d file itself. For example, if the timestamp of your header changes, it may have acquired new dependencies, and a new .d file has to be generated BEFORE make determines if the .c file is out of date.

See:

https://www.gnu.org/software/make/manual/html_node/Automatic...

The purpose of the sed command is to translate (for example):

    main.o : main.c defs.h

into:

    main.o main.d : main.c defs.h

The first line isn't correct because the .d file itself has no dependencies.

2) By the time you are compiling, it's too late to generate .d. The .d files are there to determine IF you need to compile.

EDIT: I am trying to generate a test case that shows this fails, but it seems to actually work.

Hm yes I'm convinced it works, but I have to think about why. I guess one way of saying it is that the previous .d file is always correct. Hm.

(2) is not quite correct. The old .d file from the previous compilation is actually all you need to determine whether the .c file needs to be recompiled. It works in all cases. If the .c file is new (or you're doing a clean rebuild of the whole project,) it will always be compiled, because there will be no corresponding .o. If the .c file, or any of the .h files in the old .d gain new header dependencies, they must have been modified, so their timestamps will be newer than the .o file from the last build, hence the .c file will be recompiled and a new up-to-date .d file will be generated (because a new .d file is always generated when the .c file is compiled.)

If (2) is not correct, then (1) is not needed either. The old .d files from the last compilation pass are sufficient to know which files need to be recompiled in the current compilation pass. Make does not need to know the dependencies of the .d files themselves, it just needs to load all the existing .d files at startup.

EDIT: Yep, I'm fairly confident this works :D. I don't know if whoever wrote that manual page knew about -MD, but I think it might be newer than -M, which would explain it.

Hm yes I just figured that out the hard way -- <scratching head>.

This feels hacky, but yes it seems to work. I'll think about it a bit more. (I might clone this feature for a build tool -- since the gcc/Clang support is already there, it seems like any serious build tool needs this. Although some have their own #include scanners which is odd.)

Thanks for the information!

I guess a simple way of explaining it is that if there are any new header dependencies, one of the files that make already knows about must have been modified to add the #include statement, so make will correctly rebuild the .c file and generate the new .d file, even though it's working on outdated dependency information.

Though, I guess I wasn't quite correct either. See plorkyeran's sibling comment re: generated header files.

> Although some have their own #include scanners which is odd.

I once worked in a place that had its own #include scanner (partially because it used so many different compilers and other tools... suffice it to say that Watcom C++ was in the mix). To make it work, you had to install a custom filesystem driver that intercepted disk reads during compilation and logged them. A rather... bruteforce approach. But it had the advantage of working with everything.

The problematic case is with generated header files. Suppose foo.c includes foo.h, where foo.h is generated by a separate command. On a clean build, there's nothing telling Make that it needs to build foo.h before foo.c, so it may not happen (and worse, it may usually happen but sometimes not when doing parallel builds). A separate invocation of `gcc -MM` works for this, as when it generates the dependency information for foo.c it will see that it needs foo.h before you do the actual build.

Personally I've never found it too burdensome to just manually specify dependencies on generated files.

Wouldn't the header need to be explicitly listed as a dependency to prompt it's generation anyway?

There's a -MT flag which does what the sed line does. From the gcc man page:

  Change the target of the rule emitted by
  dependency generation... An -MT option will
  set the target to be exactly the string you
  specify...

So in your example one might do something like

  -MT "$@ $(basename $@).d"

which would output

  main.o main.d : main.c defs.h

for the main.o target.

'include' being one of those features that isn't in basic POSIX make, though.

Use Rust.

Kinda sorta kidding, not-kidding. Usually I'd wrangle CMake into submission since I tend have diverse targets(Win32, Android, Linux, OSX, etc) however Rust(and moreso Cargo) makes this a pleasure to do.

Even native dependencies are straightforward unless there's some sort of build-fu going on(like luajit). You can just pull in the GCC crate, it'll shell out to clang/gcc/msvc and since you're using it in a build.rs you can configure it however you want since it's just another Rust program.

Reply on news.ycombinator.com