How to list all the targets on a Makefile | Hacker News Reader

How to list all the targets on a Makefile

diamantidis.github.io

138 pointsdiamantidis_io59 comments

When you install bash-completion on your system, usually a function is defined to autocomplete make rules for you. It works with all projects, as it does not rely on the rules being specially marked. (https://github.com/scop/bash-completion/blob/master/completi...)

Before I knew about it, I had written a similar thing:

    # Autocomplete make rules
    _make () {
        local cur prev words cword cmdline
        _init_completion || return
    
        # command to generate the rules
        words="$(make  -pRrq |\
            awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($1 !~ "^[#.]") {print $1}}' |\
            sort | egrep -v -e '^[^[:alnum:]]' | xargs)"
    
        COMPREPLY+=( $( compgen -W "$words" -- $cur) )
    }
    if declare -f _init_completion > /dev/null; then
        complete -F _make make
    fi

It relies on `make -pRrq`, that is a very useful command to debug your makefiles, especially in a big project. The option `-p` prints the make data-base, -R and -r removes the implicit rules and variables, and -q indicates only asking make a question about current state, avoid executing anything.

Well, it worked for a time, but the bash-completion one is nicer (though much, much more complex). Very nice however to know about `-pRrq`, definitely helps when working on big projects.

Keep in mind that you are referring to GNU Make specifically and won't be truly portable. BSD Make, for example, doesn't have -p:

https://www.freebsd.org/cgi/man.cgi?make(1)

There is a similar snippet by user 'cies' here on HN just 6 months ago.

https://news.ycombinator.com/item?id=21812897

This is a good point. One big problem with solutions that match the text of the makefile is that they don't catch dynamically-generated targets.

I have makefiles used for data-crunching that are parametrically-generated using $foreach in gnu make. One typical use case is "make these plots for all the data, for 10% of the data, or for just one example" - with targets generated by a $foreach(...) construct.

Asking make to regurgitate the targets it knows about catches these guys, grep doesn't.

There are many other examples of dynamically-generated rules, such as using macros that create rules via gnu make's "define".

Here's zsh's implementation: https://github.com/zsh-users/zsh/blob/master/Completion/Unix...

It's more complete, since it implements completion for bsdmake too, where there's no -p but a comparable debug option.[1]

It completes variables in addition to targets, too.

[1] https://www.freebsd.org/cgi/man.cgi?make(1)

You want this:

  apt install remake
  remake --tasks
  remake --targets

The "remake" tool is "GNU make" + some extra bells/whistles, most notably a REALLY usefule interactive debugger. It's 100% compatible with GNU make because it is GNU Make. And you can ask it for --tasks and --targets.

The `sed` and `grep` syntax isn't always portable to non-Gnu versions of the tools. The following version doesn't use gnu-isms for `grep` and `awk` for and works on both BSD's and MacOS, sorting and printing comments printed after the ##. Does require on Gnu make for $(MAKEFILE_LIST), but the filename of the Makefile can be used instead:

    help: ## Print help for each target
         @grep '^[[:alnum:]_-]*:.* ##' $(MAKEFILE_LIST) \
         | sort | awk 'BEGIN {FS=":.* ## "}; {printf "%-25s %s\n", $$1, $$2};'

The article recommends `.DEFAULT_GOAL := help` which causes make to display the help message when the user doesn't specify a target.

But this breaks the classic incantation:

    ./configure
    make
    make install

if i'm remembering correctly, and autoconf-style project guidelines have the default target as 'all', so 'make all' for the second step should usually work

While we're at it, a small tip:

  make -p

(or `make -pn` when you're inside a directory with a Makefile and you don't want to trigger the default target as a side effect) lists all rules including implicit ones, which is great for discovering useful builtin variables. For instance,

  $ make -p | grep '^LINK.c'
  LINK.cc = $(CXX) $(CXXFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
  LINK.c = $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
  LINK.cpp = $(LINK.cc)

'make' is the language that I hold up as an example when I say, "When you are developing a new programming language, do the debugger EARLY because you're going to need one anyway."

Debugging multi-million-line makefiles is not fun when all you have is echo, and one shot every few hours at a race condition that hopefully your debugging output won't perturb.

What kind of a workload would require a multi-million-line Makefile? Honestly curious...

Up until a few years ago, all of android was built with thousands of makefiles, totaling millions of line easily

A common disease, I think. The builder we used in Xbox did this. It was a real pain in the fundament when things went sideways.

This feels like functionality which should be handled within make itself.

The world has too many 'make' replacements. I feel like we just need an incrementally improved make. GNU Make has enough inertia that it isn't going away, but making it slightly easier to use while not confusing old folks like me seems like a win for the community.

> This feels like functionality which should be handled within make itself.

make -p gets you mostly there.

Just type “make <tab>” in fish, because it does it out of the box :)

Here are the full completions: https://github.com/fish-shell/fish-shell/blob/master/share/c...

"make <tab> <tab>" in bash ...

I have another version based on awk[1], which instead of listing all targets, it will only list targets with a help message in the format of `@: #`, e.g.

    install:
        @: # Install the application to the given PREFIX.
        @: # Unless specified otherwise, PREFIX defaults to /usr/local

which will generates

    Usage: make TARGET
    
    You must run make with one of the following targets to continue:
    Run make VERBOSE=1 TARGET for verbose log.
    
    help:
            Print this help

    install:
            Install the application to the given PREFIX.
            Unless specified otherwise, PREFIX defaults to /usr/local

It doesn't read from MAKEFILE_LIST, though.

[1]: https://gist.github.com/sirn/6a2f4c0b1cb917ef54b1fb813330d8e...

https://stackoverflow.com/questions/4219255/how-do-you-get-t...

Doesn't this break if your tasks have dependencies, in that it'll list all the dependencies together with the task name?

Hey!! No, the regex on the sed command takes care of it. It breaks the target into three parts. One before the ":", one between the ":" and the "##" and one for the rest, after the "##". Then it outputs only the first and the third part

I remember an article describing exactly this a few years ago. Did you game up with this yourself? If not a source might be nice.

I've done something similar with descriptive summaries of each target. It's an idea that is easy to reinvent independently.

bash-completion does that for you by just pressing TAB. All I miss from it is some kind of annotation.

When your make targets are mostly PHONY targets, you may want to replace make with (bunch of) shell script(s).

Honestly, who came up with that `make run` nonsense? I see this quite often in the pythonverse.

What's wrong with using make this way? What would I win if I switched to shell scripts?

One thing you win is that the name of the script can give you a hint as to what it does.

You avoid all the complications of make. Extra syntax, variable substitution, lack of functions and loops, ...

Why use a bunch of shell scripts when you can use one file with a traffic cop built-in?

Traffic cop?

GP might be referring to "make -j 4" which allows controlled parallelism.

Yes, also parameter directs execution to the correct subscript if/when needed. No conditional boilerplate needs to be written, arcane enough in shell script as to need looking up every time.

Make is kinda weird too but the complexity appears later. The traffic direction is trivial.

Good stuff! Regrettable that GNU make has no command-line flag for this IMO.

Simple but really effective trick. Brian Kernighan would be proud :)

With zsh or fish, just press "tab".

TLDR: grep

tldr make tab tab?

Makes me realise how nice “npm run” is. Use it loads.

It's not even comparable to make.

I absolutely despise make and think it's ridiculous that a tool that can't handle spaces vs tabs is so "mainstream".

Most newer/younger people don't like or know Make; it's old, arcane software.

It differentiates between TABs and Spaces because each one signifies a different thing to follow: Lines starting with TAB indicate that a _recipe_ follows (which is shell syntax), and lines NOT starting with TAB indicate that a _rule_ follows (ie., native Makefile grammar/syntax).

Hate it or not, it's been a reliable workhorse for decades.

Disclaimer: I like make(1).

That said, the description above is accurate, but still describes an almost immediately regrettable mistake[0]. Many are used to it now, and the semantics are “fact” now, but could still have been avoided.

[0] https://stackoverflow.com/a/1765566

Thank you for the epic reference! Despite all the classic works in computer history, I've come to realize that TAOUP is honestly/possibly my favorite. ok, maybe tied for favorite...

If THAT's your beef with Make, a pro tip: you can use ";" instead of TAB. Problem solved?

If your makefile has so many targets, maybe you are doing too much and you need to use another tool such as autoconf/automake on top?

Autotools projects tend to have recursive makefiles. Look up "recursive make considered harmful" for a good explanation of why that's bad. Edit: https://pic.blog.plover.com/prog/lib/auug97.pdf

Then, the actual benefit that autotools provides, which is portability for C code running on mostly Unix-like systems, is a pretty dead topic today. I am not talking about all the C bashing people do in a place like HN, what I mean here is that C99 and POSIX compliance is pretty universal now in a way it wasn't 20 years ago. And there are much fewer Unix-like systems in common use. (Linux has taken over most.) Nobody has to run a check for whether it's safe to include string.h or the return type of getpid() anymore.

ok, makes sense.

But then why would somebody have so many targets on a makefile? What are the common scenarios for having so many targets in a makefile if not for doing different installation options?

That's like saying "why are there so many shell scripts?" Make is a generalized scripting environment.

So in one project you might have a target that builds an app, a bunch of rules that generate manpages, another that puts it in /usr/local, another that code signs, another that uploads to some app store.

In the systems I have seen:

  * Because the rules are generated. Building a subpart sometimes means invoking a generated rule.

  * Because make is used for other things: building binaries, libraries, examples, test suite, documentation -- then executing them, preparing a test environment, installing them, generating distribution packages, etc.

  * Because the system is a framework, and each rule relates to one of many plugins (Buildroot for example).

  * Because the system is just very large and complex (linux).

Now to go back to automake. It is still very useful to have automated checks of compliance -- either about some extended warnings supported only by some compilers and not others (clang vs GCC), allowing extensions, static analysis, coverage, etc. That being said, automake is terrible. The two passes to generate the actual build systems means most people are actually reading generated makefile and debugging that. As was said by GP, it is also sub-optimal regarding use of make.

Frankly, it is just much simpler to directly work with makefiles, or change build tool altogether (meson + ninja is nice). Autotools has always been a nightmare to work with in my experience.

Keeping with make, I much prefer musl way for example, where a POSIX shell script examines the dependencies and configures a single makefile giving the project state, then the actual makefiles will consume it to operate properly.

> Frankly, it is just much simpler to directly work with makefiles

agree.

pmake (colloquially bsd make though it arrived late in the CSRG era) allows for the concept of a 'makefile library' that handles common things (e.g. '.include <bsd.prog.mk>' to build a program if the project directory is laid out appropriately) which I think would have been a much better approach for autotools instead of generating makefiles in multiple stages. Unfortunately the autotools approach is many peoples 1st exposure to anything make-like so they are left with a bad impression of makefile complexity.

IMHO pmake also has a nicer and more flexible syntax extension over posix vs gnumake (e.g. can define a dynamic target in a for loop). Unfortunately pmake is not very well known outside of the BSD community.

Reading and debugging the generated makefile should not be the norm IMHO. For the same reason that is isn't usually required to read and debug generated assembly (from C), Java/Python bytecode etc.

Sure, except that the language used to generate the makefile is substantially harder to read in a lot of cases. And it doesn't tell you important things such as how libtool paths are configured or what paths are included in the gcc commands. These aren't that important unless you're cross-compiling for a system that doesn't have the same libc, libgcc, or library set and for which you need to set a "sysroot." Different projects have different ways of doing this, and sometimes a project will use something that sets sysroot appropriately for libtool but then doesn't generate the --sysroot command for gcc.

For historical reasons Cross Compilation against a different root filesystem for a different target architecture is somewhat of a second-class citizen in most automake/autotools build systems and you really do have to read the makefile output to figure out how they've configured their particular instance.

A Makefile that's basically a tool with multiple entry points, like `git'? The one I've got for my current work project has 36 such targets. Some people might have an individual shell script for each one, but this is just what I prefer to do. Everything's conveniently in one file, and it's easy to share code between targets.

The convention I use is for all entry points to be phony targets. Non-entry point phony targets have name starting with _. The helper I can then use is:

    @grep -E "^\.PHONY:" Makefile | grep -v ":default" | sed "s/.PHONY:/  /" | grep -v -E "^[[:space:]]*_" | sort

(The two spaces that replace ".PHONY" are just indentation, so the list stands out a bit more visually when scrolling back.)

The phony target named "default" is the default one, that runs the above command, so I don't need to see its name.

I've recently started using Make for my Python project as a convenient way of running arbitrary code checks. I like having targets like the following:

test - runs pytest with whatever config flags are needed coverage - run test and then check for minimum code coverage run - run the application with sane defaults mypy - does type checking (mypy) lint - run flake8

And I usually have `make all` setup to run everything (static analysis before tests, then code coverage checks).

In another project, I built out some testing infrastructure for a Rust project (setting up a testing database and tearing it down at the end).

I sometimes feel like I "should" use a dedicated tool for this, but Make is ubiquitous and easy enough to use for simple use cases like this.

In my experience, with very large software projects, the multiplicity of targets tends to be related to a combination of multi-stage builds (where you need to build some tools to generate files that are needed to compile the project), installation insanity (e.g., needing to build a large set of locales for upload to the FTP server), and testing (differentiating between multiple test suites!).

Make is not just a build system; it's also an interactive development tool. It's useful to be able to build parts or variants, run tools such as static analyzers, build and run unit tests, flash embedded devices, etc. Make can orchestrate all that without duplicating the effort to track source lists, dependencies, platforms, etc.

Here's a typical example that happens to be my current hobby project. Take a look at the first block comment and the "help" target.

https://github.com/kbob/MinimumViableSynth-5/blob/master/mak...

https://www.microsoft.com/en-us/research/wp-content/uploads/...

Heh, when I went to search for what GP recommended, this apparent rebuttal was the first result.

Reply on news.ycombinator.com