Afraid of Makefiles? Don't be
matthias-endler.de
matthias-endler.de
1. Using make in a CI system doesn't really work, because of the way it handles conditional building based on mtime. Sometimes you just don't want the condition to be based on mtime, but rather a deterministic hash, or something else entirely.
2. Make is _really_ hard to use to try to compose a large build system from small re-usable steps. If you try to break it up into multiple Makefiles, you lose all of the benefits of a single connected graph. Read the article about why recursive make is harmful: http://aegis.sourceforge.net/auug97.pdf
3. Let's be honest, nobody really wants to learn Makefile syntax.
As a shameless plug, I built a tool similar to Make and redo, but just allows you to describe everything as a set of executables. It still builds a DAG of the dependencies, and allows you to compose massive build systems from smaller components: https://github.com/ejholmes/walk. You can use this to build anything your heart desires, as long as you can describe it as a graph of dependencies.
Highly recommend reading https://ocw.mit.edu/courses/electrical-engineering-and-compu... for some theory on parallel execution for graphs, if your interested in things like this.
Any CI I've seen starts from a fresh repo checkout and rebuilds everything every time, so it's not an issue with practice.
OTOH, probably all my projects were small enough for it to not hurt when the CI builds everything from scratch every time. I might look at this from another angle if the things I worked on were mega-LOC C++ programs, and not kilo-LOC Go programs.
By the way, I just noticed that compilation times are another argument for microservices.
Honestly, if I were to modify make for some custom behavior, I would look for another tool¹ or create yet another builder tool. Because although I already know most of the syntax, point #2 is an incredibly big problem, the GP missed point #4 in that make does not manage external dependencies², and the syntax is not bad only for learning, but for using too.
1 - Almost everybody seems to agree with me on this. The only problem is that we have all migrated to language constrained tools. This is a problem worth fixing.
2 - Wether it is installing them or simply testing if they are there, make can't do any. Hello autotools, with it syntax set to extreme cumbersomeness.
Make is about "making files." Or to be a little more semantically specific, Make is about "processes that transform files to make new files based upon their dependencies". It really doesn't matter that your file is a C source file, or some unpreprocessed CSS, or a template, or an AMI. To your AMI example, if you specify a dependency (or dependency chain, DAG) as a time-stamped (or set of) file, you can get make to rebuild the AME for you along with any other supporting or intermediate files.
IMO, the suckless guys are masters at writing deceptively complex but highly readable & concise Makefiles. Here's plan9 workalike utility library [0], a util-linux/busybox package of binaries [1], a window manager [2], a terminal emulator [3], and webkit2 based web browser [4]. I highly recommend you study them if you're looking to up your Make game.
[0] http://git.suckless.org/9base/tree/Makefile
[1] http://git.suckless.org/sbase/tree/Makefile
[2] http://git.suckless.org/dwm/tree/Makefile
I'm not saying you can't use make (we did use make before we switched to walk), it's just more painful for non C/C++ build systems. All we really want from a build systems, is a generic tool for describing, and walking, a graph of dependencies.
If you're careful (and even if you're not), loops in your dependency graph are usually a non-issue. And if you use a single `Makefile` it'll detect the circular dependency and try to ignore it.
Let's take an example processing Sass CSS files. You just need 2 folders in your project, `css/` and `scss/`, and a `Makefile` to process all your Sass files into CSS.
# Locate our source assets, save a list
SRC = $(shell ls scss/*.scss 2>/dev/null)
# Specify a filename transformation from scss to css and convert the SRC list
OUT = $(SRC:.scss=.css)
# Define rule of how a scss to css transformation is supposed to happen
css/%.css : scss/%.scss
sass $<:$@
# Make the `all` target depend on the OUT list
all : $(OUT)
If you use @import in your scss file the css will not be rebuilt if only the imported file is modified.
So no. It does not take 5 lines to teach make how to build Sass. To solve this you either need to teach make about sass @import, or teach sass about make and let it generate makefiles (like gcc -MMd). Or simply just use sass --watch for incremental builds and take the recompile hit if you need to restart it for whatever reason.
(While I'm at it...Another thing missing from that makefile is source maps (.css.map) generated by the sass compiler. It's not only one css file being generated from each sass file. That will complicate the rules even further)
And yes, by that logic, `SRC = $(shell ls scss/*.scss)` is dumb. That is not lost upon me.
To fix the `.css.map` issue is simple. You can't use the pattern you've seen for yacc/bison, though. The `$@` var would have both files in it.
%.tab.c %.tab.h: %.y
bison -d $<
css/%.css : scss/%.scss
sass $<:$@
css/%.css.map : scss/%.scss
@touch $@ css/%.css : $(SRC) scss/%.scss
sass $<:$@
css/%.css.map : $(SRC) scss/%.scss
@touch $@Or teach a third-party tool to output a list of @imported files for a given Sass file. Then this list can be used as that Sass file's dependencies.
I meant "in practice", of course...
I haven't found that to be the case. Once I show people how simple it is they realise make is somewhat approachable, similar to OP's article.
>walk is currently not considered stable, and there may be breaking changes before the 1.0 release.
:)
It also needs go, which is pretty non-lightweight dependency on windows, I guess. (And personally, I don't find these dir-hierarchy connection and bash's "case $target in a) ... b) ..." any friendlier at all, though make has some quirks with variable interpolation.)
I'm not to argue to use make for very complex situations, but usual src->intermediate->executable and generate this from that of any size and count is a perfect task for make. Makefiles are unsuitable not for big projects, but for complicated build systems, where it's not enough to just do them steps in correct order. If your build system is complicated, it should be worth it at least. Otherwise use make.
Fix: typos/grammar
If you don't like bash syntax, you can use Python, or Ruby, or Perl, etc. Any executable can serve as a Walkfile.
It's a completely fair point that make is installed on pretty much every machine by default, which is why we won't see it going away anytime soon (nor should it, it's still good).
Also be sure to have a look at tup, which operates vastly more efficient by simply walking the DAG in the opposite direction:
That is, instead of looking what you want to build and checking all timestamps of dependency files, it can use e.g. inotify, then know exactly which files changed, and rebuilds everything that depends on those files.
Moreover, it performs the modification check only once, at the beginning. During the build, it doesn't need to re-check everything, because it already knows which files were recreated.
I believe that tup does have some features in that direction, but I may be mistaken.
I have foo.c, bar.c and a rule like:
: foreach foo.c bar.c |> ^o^ gcc -c %f -o %o |> %B.o {objs}
: {objs} |> gcc %f -o %o |> baz
I used this with some of my literate code projects where I had tup running org-tangle for me (via an emacs script). If I'd only updated the documentation, and the code hadn't changed, nothing else would build. If I'd only changed unimportant parts of the code that generated the same object files, no new binary or library would be built.
Maybe someone who used tup more recently can pitch in
This is my #1 gripe with Make, and many other build systems as well. There are so many flaws in the timestamp approach. Most are easily fixed with cryptographic hashes.
I like the OCaml build system OPAM for that matter, it internally just stores checksum. I believe it also uses timestamps to speed things up, but only for equality comparison (not for older/newer comparisons which may easily lead to wrong result).
But that's kinda silly.
I might argue in favor of (fast) cryptographic hash algorithms in general since they're fairly well understood / implemented / hardware accelerated / tend to have extremely "balanced" random output thus less likely to accidentally conflict... but that's about all I can think of.
Note that "recursive make is harmful" does not argue against multiple Makefiles!
There's nothing wrong with using multiple Makefiles per-se, as long as they include rather than call each other. In other words, the article just says that Makefiles should use sub-Makefiles via "include" rather than executing those through separate (recursive) call to Make.
However, I agree that composition of Makefiles is still a pain, given that the included Makefile must be aware that it is executed from another (parent/grandparent) directory.
Your README’s example show `parse.h` as output from `Walkfile deps parse.o`, I think that is a mistake.
As for your build system (and comment), I have some questions:
1. How do you achieve using a deterministic hash as condition (and aren’t all hash functions deterministic)?
2. Why would you not be able to use mtime as a dependency? The only case I have run into is when the build depend on data from remote machines, but in that case, I think the proper solution is to have an initial “configure” step where you retrieve the data your build depend on and/or a build rule to update this data.
3. Does your build system execute the Walkfile for every node in the graph on each walk? Because that sounds like a quite noticeable overhead for larger projects.
4. Am I right in thinking that the primary advantage with your system, over make, is that a shell command is executed to obtain a target’s dependencies?
2. Mainly because doesn't translate across machines; it only applies to your machine. If someone checks out the repo on their machine, mtime is different. As soon as you move a build system to CI, you need something better than mtime, like content adressable hashes, if you intend to cash targets.
3. It executes the Walkfile twice for each target in the graph; once to determine the targets deps, and once to execute the target. This definitely hasn't been a bottle neck for anything I've used walk(1) with so far.
4. Correct! But even more so, replace "shell script" with "executable". The Walkfile can be written in any language you want, as long as it has the executable bit set.
As for #1, what I do not understand is how a `Walkfile` allows me to use a content hash change to trigger a rebuild.
Your documentation says that a file list should be returned for `deps`, so how does the `Walkfile` communicate that e.g. `main.o` should be updated if `sha1sum main.c` is different than on last invocation?
That's a trivial example using mtime. You could replace it with a hash function if you wanted.
This is one of those things that "distcc" and "ccache" have dealt with effectively - anyone trying to build big C++ projects would be well served by looking at those tools.
Multiple Makefiles works fairly simply with the include directive.
The thing I've found painful before, with C projects, is getting Make to recognise that it needs to rebuild when a header has changed. There are various ways around this (makedepend etc) but they've all been quite painful to set up and not quite perfect.
That said, with modern machines that have NVMe storage and massively fast processors, a complete rebuild is seldom a big time cost
http://make.mad-scientist.net/papers/advanced-auto-dependenc...
You already tried that? I do not find it painful to set up.
The syntax is a tad hairy though,and I'd want to adapt it - I tend to avoid compiling individual C files to objects these days, due to WHOPR optimisation.
That said, I think incremental builds are important for most use cases.
And btw there is no way to use make without shell, but you can use shell without make.
Sure, I can write these as functions in bash, call once for each source file, check return codes etc etc, but I find expressing dependencies faster than writing anything like robust code in shell, and make deals with failed steps by stopping.
On the other hand, computers are fast enough that serial full builds indeed work in some cases.
Are Android and LEDE/OpenWrt big enough?
> 3. Let's be honest, nobody really wants to learn Makefile syntax.
That's probably very subjective, I find JavaScript, C++, PHP or Perl much worse :-)
Anyway, for the past years I'm cheating a lot and using CMake to get complex Makefiles almost for free.
Oh definitely. I actually like Makefile's, but in my experience, more teams have a deep familiarity with some programming language, than with Makefile syntax. I haven't met very many people who have read the GNU make manual, and know all the idiosyncracies around Makefile syntax.
target/linux/ar71xx/image/Makefile: CMDLINE = $$(if $$(BOARDNAME),board=$$(BOARDNAME)) $$(if $$(MTDPARTS),mtdparts=$$(MTDPARTS)) $$(if $$(CONSOLE),console=$$(CONSOLE))
If BOARDNAME is defined, add board=$(BOARDNAME) to the string
If MTDPARTS is defined, add mtdparts=$(MTDPARTS) to the string
If CONSOLE is defined, add console=$(CONSOLE) to the string
You must have seen more complex clauses than that in shell scripts and all sorts of places.
It feels much more like in those IDEs where you just drag all the source files in and you're done.
I even prefer editing visual Studio project files by hand to many makefiles out there..
Doesn't mean that it's not hard. :) And if IIRC google are trying to move android build to blueprint because make is too complicated.
`unzip -D` is your friend.
My makefiles are usually just a way to record a data pipeline. Get these files, shove them through these scripts here and those programs there. Launch a web server to show the output.
Make's syntax is quite simple. It's a bunch of variables one has to memorise, and man page is a command away. And any decent editor would know to insert literal tabs.
> Make is _really_ hard to use to try to compose a large build system from small re-usable steps.
I have no experience myself, but Linux's build system is a bunch of homebrew makefiles, all the BSDs and their ports trees build with bmake. These are enough positive examples for me to think that Make is good for big systems.
I'll just leave this here:
https://github.com/sapcc/limes/blob/62e07b430e2019a6c1891443... (then used in line 42)
Only if each Makefile is treated as a separate rule set processed with a separate make invocation.
> Read the article about why recursive make is harmful
That (now) venerable, old paper in fact shows how to break up into multiple makefiles (called "module.mk" in its examples) which are included into one graph.
(It's possible to actually have this file be called Makefile. Not only that, but it's possible to have it so you can actually type "make" in any directory of the project where there is a Makefile, and it will correctly load and execute the rules relative to the root of the tree.)
a) Mtimes-as-change-detection is fundamentally broken given the reality of networked file systems and... physics. (Minor problem, but extremely annoying to work around in practice.)
b) Nobody can actually really understand all the interdependencies between all the code in their system(s!), and yet Makefiles expect you to specify all of that explicitly?!? Yes, you could technically specify that -- and you'll want to -- but you won't, because you don't know and don't have the time.
c) Make support for builds that change the structure of the build during the build is abysmal. E.g. after "processing foo.xml we now have more files than we had before! What are you going to do?". Well, in Make it's some sort of custom solution with ".d" files and "gcc -M" or whatever. This is utterly broken in that it pushes all the complexity onto the user. So, yes, an elegant model, but it doesn't actually solve the problem. If you want to see a better solution see the "Shake" paper.
A properly hand-tailored Makefile is a thing of beauty, and it is not difficult.
https://github.com/devkitPro/nds-examples
They're very clearly hand made, do just enough to be immensely useful, and are well commented enough to be easy to modify for a Make beginner, just like I was back in the day. I'm sure there are other great examples floating around the 'net, but I cut my teeth on GBA and NDS development, and taught myself make by following in their footsteps.
Meson seems to be the latest fashion in Linux circles, it seems...
You are trying to solve a Turing complete problem in a simplest low impact way as possible.
There is no good solution. Everyone will fall short in one way or another. Hence a new build system. The wheel never stops.
Most build tools do have some kind of Turing-complete tooling built in with their DSL. But I don't think that's an absolute necessity.
These symbols are artifacts/files. Elementary recursive Build-Scripts can be Turing complete even if the overall language isn't.
I've seen failures of such schemes to rebuild stuff when command line parameters (defines, environment) changed.
It also brings all sorts of trouble in parallel builds as Make is weak at handling dependencies that are not generated in the exact Makefile you run. (Thus non-recursive Makefile which still fail and have other warts.)
Even cmake and autoconf generated Makefile have trouble with complex projects. (Part of the reason why cmake can now generate Ninja files instead.)
Recursive Make Considered Harmful
I have a recursive-make project at the moment that suffers from none of the problems described in the paper. We will likely move to an include-based scheme before long, which will take some minor tweaking of targets in the leaf Makefiles.
Also - Considered harmful essays considered harmful - http://meyerweb.com/eric/comment/chech.html
I'll write Makefiles by hand for small C/C++ projects, but for anything serious I'll use cmake/etc.
Source: We made a "properly hand-tailored Makefile" for Rust. It started out short and elegant, but it quickly grew into a nightmare lasting 4 or 5 long years. Only about one or two people (Alex Crichton and Brian Anderson IIRC) had any clue how it operated. Proper cross-compiling support involved multiple levels of nested variable expansion all over the place to find the right build/host/target compilers so that Canadian cross builds worked. Alex ended up doing some heroic effort to throw away all the Makefiles and going to (effectively) a custom build system using Cargo, which was a massive simplification.
Then again, as a ROS user, I'm also a defacto CMake expert, so perhaps I underestimate how difficult CMake is.
I only came into KDE when the switch to CMake had already happened, and remember it as reasonably approachable (even if quirky). Most developers were familiar with it and actively authoring the CMakeLists.txt files for their own projects. (That doesn't mean that there weren't some experts again, but they focused on implementing reusable modules that the others could easily integrate into their own build system.)
my_function("${MY_VAR}")
The other case is builtin macros that expect to be passed the name of a variable that they themselves are manipulating. For example the list and string operations. See: https://cmake.org/cmake/help/v3.0/command/list.html
Most hand-tailored Makefiles have seen aside from well known OSS projects (Linux Kernel, suckless stuff) generally had major issues.
Here is a sample of what I've seen:
* not supporting common variables, specially DESTDIR in install target, but also PREFIX, BINDIR, etc, or CC and CFLAGS for compilation. (https://www.gnu.org/prep/standards/html_node/Directory-Varia...)
* bad ordering, at least preventing parallel build (-j N)
* bad error handling which result in silent failures, I've seen it in Makefiles using for/while loops, if/then/else, or successions of commands (cmd1;cmd2) inside targets for example.
Makefiles are also a little too crude for somewhat complex projects. Doing things like searching header pathes, detecting which OS you're on, what is the CPU endianness, searching the required dependencies and recovering information such as version about said dependencies would be extremely painful to do with only plain Makefile.
CMake contains a lot of ready to use modules to tackle the most common issues, and you can easily write your own modules if you need to.
Aside from very simple project, I would not recommend using plain Makefiles, there are just to easy to do wrong.
...but scaling from the small to the large elegantly is what make doesn't do well; and there's an astonishing number of tools designed to try to work around the problem.
A lot of people hate CMake for its strange language and (arguably) questionable design choices; but it scales to large projects without significant problems; you only have to look as far as the android native client makefiles to see how the truely heroic efforts to use make have resulted in makefiles that are only... moderately terrible, instead of absolutely aweful.
I feel like there's this nostalgic desire from many programmers to 'embrace simplicity' and avoid the complexity and annoyance of learning and using complex 3rd party tools when 'simple' solutions are good enough.
...but often those simple solutions are ever any good at a trivial scale.
Then we started using maven, and man, maven is ridiculously complex, especially adding custom tasks, but at least it was declarative. After getting into Rust, I have to say, Cargo got the declarative build just right.
But then, for some basic scripts I decided to pick Make back up. And I wondered, why did we move away from this? It's so simple and straightforward. My suggestion, like others are saying, is keep it simple. Try and make declarative files, without needing to customize to projects.
I do wish Make had a platform independent strict mode, because this is still an issue if you want to support different Unixes and Windows.
p.s. I just thought of an interesting project. Something like oh-my-zsh for common configs.
https://www.gnu.org/software/make/manual/html_node/Guile-Int...
https://jaxenter.com/gnu-make-4-0-adds-guile-output-sync-bre...
FWIW I wrote about how Make, shell, and awk heavily overlap in functionality here:
http://www.oilshell.org/blog/2016/11/14.html
This probably won't happen for quite a long time, but I'd like to expand my Oil shell to have the functionality of Make.
On the one hand, it's kind of ironic that you're asking for Tcl, when shell is Turing complete and already such an integral part of Make (every line of the Makefile spawns /bin/sh).
On the other hand, I understand that shell is a language with many sharp edges and most people dislike it. The point of Oil is to get rid of the sharp edges, and then maybe shell can take the place of Tcl/Guile.
It seems a little ridiculous to write build scripts in make, shell, and guile, all of which are Turing-complete (not to mention Awk or Perl, which often show up). And I don't actually think Lisp/Scheme are very good languages for Unix-like text processing and syscalls/OS integration.
Absolutely love your oilshell posts. Keep it up. I think a concatenative language would be great as the embedded logic for Make, but a Lisp would also work.
Combining make and shell would mitigate this problem. Then you would need one binary in sync instead of two. Well, I suppose you also need "busybox", e.g. cp, mv, mkdir -p, etc. But in practice I think that is less of an issue than shell and make colliding.
Thanks for the encouragement! (If you didn't catch it I linked these build system observations in a sibling comment: http://www.oilshell.org/blog/2017/05/31.html)
I've played with concatenative languages, and read a lot about Forth. I'm not sure I see them as great for "logic". They are very elegant for certain problems, but fall down for others (IIRC the quadratic formula was a popular example.)
The main kind of logic you need in Makefiles is expressing build variants -- e.g. for debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc. I think that can be done fairly well with a Python-like imperative language with dicts and lists. (Some people have mentioned Bazel, and it is derived from Python and works fairly well for that.)
Write portable shell scripts (or lines, in the case of a Makefile).
There's some tips here: http://people.fas.harvard.edu/~lib113/reference/unix/portabl... and of course you can refer to http://pubs.opengroup.org/onlinepubs/009695399/utilities/con...
There's no way to test that the commands you're running will actually work on someone else's machine, other than by "memorizing the manual".
I have an entire book on this:
https://www.amazon.com/Beginning-Portable-Shell-Scripting-Pr...
It goes through all the commands and common versions of Unix and which flags are likely to work, etc.
Even that book admits there's a lot of folklore, because nobody has actually gone and tested things recently. Something as simple as safely writing with "echo" is a problem. You can argue that any script that uses echo $foo is incorrect (because $foo might be a flag). Conversely 'echo --' is supposed to print -- by POSIX.
The only people who are likely to even attempt this are people whose full-time job it is to write shell scripts, or the authors of tools like autoconf, which must generate portable shell. autoconf shell is a good example of the anachronisms and hoops you need to jump through to support this style.
Nobody else has time for that, because they have to spend their mental energy writing C and C++, not shell and make. So that's why we have pretty low standards for the quality of build systems. The tools aren't there to support writing a robust and easy-to-use build system.
There is way too much incidental complexity in /bin/sh and almost all build systems. Unix is awesome, but I also hate Unix for being "good enough". Unix Hater's Handbook and all. I hate it from above, not below.
We should always be seeking to reduce the cognitive burden in the tasks we do. One level of fail for /bin/sh is the level of semantic density and the lack of discoverability. It violates the principle of least surprise like nothing else I have used. Look at variable assignment!
export FOO2= "true"
export FOO3="true"
The biggest problems with sh and Make are that lack of debugging and introspection. Any follow on tool that would displace them should make developer ergonomics the highest priority.
When Bash is invoked as "sh", it runs in posix compatibility mode.
There is a reason some of us keep harping on about "don't write bash-scripts, write portable shell scripts".
> I despise /bin/sh, it is basically PHP.
Similarly, I despise chickens. They're basically the sub-prime mortgage crisis.
As a learning experience, I wanted to implement parts of the GMSL[3] on top of guile. It looked like it could be useful to others, so I made it into its own project: the GNU Make Toolkit[4] (it's still very much a work in progress - I've put more effort into the test suite and the documentation generator than into actual functionality).
[1]: http://git.savannah.gnu.org/cgit/make.git/tree/guile.c
[2]: http://git.savannah.gnu.org/cgit/make.git/tree/gmk-default.s...
The CMake scripting language [0] .
But then again maybe that's just an argument against C and C++'s compilation model. :)
Yes, that's annoying and definitely a problem with the inclusion-based dependency model of C. But really it's not such a big deal.
This is not as simple as it sounds. Make has a fundamental limitation that dependencies cannot be generated on the fly, and current workarounds for dependency-generating dependencies are very messy [1].
[1] http://make.mad-scientist.net/papers/advanced-auto-dependenc...
Not so fundamental, if you quickly make depend after changing includes. Why is that messy? I never got that "buy magic automation, sell simplicity" trading. Deps changed, deps have to be updated. Once updated, they just work.
* http://jdebp.eu./FGA/introduction-to-redo.html#CompilerDefic...
Almost all other systems have this flaw in some way as well. You can fool everything, because a system capable of not being fooled would literally have to reparse and rebuild everything from scratch.
http://news.dieweltistgarnichtso.net/posts/redo-gcc-automati...
Also, have you looked at the redo tool redo-ifcreate? http://news.dieweltistgarnichtso.net/bin/redo-sh.html
You've switched tack between "not such a big deal" and "can't be solved" in the space of two messages. Both of your extremes are wrong. The truth is that this is simply difficult and needs either improvements to compilers or, as in my case, add-on tools that replicate the compiler's pre-processing and emit both the redo-ifchange and the redo-ifcreate information.
> In 2014, Nils Dagsson Moskopp re-implemented Pennarun redo, retargetting it at the Bourne Again shell and BusyBox.
I targeted the Bourne Shell (sh), not the Bourne Again Shell (bash). Also, my redo implementation contains redo-dot that paints a dependency-tree – I have not seen this otherwere.
These include amongst others local, $(), >&-, the -a and -o operators to test, and command. (You also failed to guard against variables expanding to operators in the test command, but that is a common general error rather than a Bashism.)
Beware of testing against /bin/sh, even the BusyBox one, and assuming that that means POSIX compliance, let alone Bourne shell compatibility. Even the OpenBSD Korn shell running as /bin/sh or the Debian Almquist shell running as /bin/sh will silently admit some non-POSIX constructs.
I plan to work on POSIX compatibility for my redo implementation.
The real problem I have is that make generally relies on shell commands like cp, so there's no truly cross-platform (e.g., Windows and POSIX) way to copy files around in a makefile. I guess you could build your own utility, use it, and delete it when you're finished building. I've actually resorted to using ExtUtils::Command ( http://perldoc.perl.org/ExtUtils/Command.html ).
But because it is Make written in Java with a XML syntax, it has inherently the same problems.
Fun fact: They had for years a makefile rant on their home page which boilt down to "Ant was invented because its developer couldn't make Makefiles work as his editor didn't properly show tabs vs spaces": https://web.archive.org/web/20100203102803/http://ant.apache...
Still ugly and terrible and limiting.
You declare targets and dependencies and the steps within are target a executed procedurally in sequence.
This sentence is 100% true for both Ant and Make.
Ugly, yes. It was developed during the times when everything had to be XML. But also Makefiles don't win a beauty contest.
Terrible. Maybe, but not worse than trying to write portable Makefile for something non-trivial (e.g. libraries).
Limiting? In what way? You are encouraged to use the supplied tasks which cover a lot. Creating new tasks would be done in the language you programming. You can still execute any arbitrary command if you really want to.
Ant is declarative in that you do not directly refer to files except in tasks that directly handle files. You cannot say that phony task x depends on file y.
I was stuck on an old DoD redhat box and it didn't have gnu parallel or other such things and co-worker suggested make. It was available and it did the job nicely.
If you need more power, use the wildcard expansion and "patsubst" type rules to define rules at runtime.
Sometimes Redhat backport things but I just remember not finding anything there and then being very happy to have discovered make. I think even with xargs -P I would have gone with make anyway as it involved a few dependencies and checking of file times.
I use a build system, redo rather than make, to rebuild my GOPHER site and Debian/FreeBSD package repositories. (https://news.ycombinator.com/item?id=14837740 https://news.ycombinator.com/item?id=14928216)
Also see this if you have a system without GNU Parallel: http://oletange.blogspot.dk/2013/04/why-not-install-gnu-para...
From memory here's a Makefile that serves most of my needs (use tabs):
SOURCE=$(wildcard *.c)
OBJS=$(patsubst %.c,%.o, $(SOURCE))
CFLAGS=-Wall
# define CFLAGS and LDFLAGS as necessary
all: name_of_bin
name_of_bin: $(OBJS)
$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)
%.o: %.c
$(CC) $(CFLAGS) -o $@ $^
clean:
rm -f *.o name_of_bin
.PHONY: clean all CFLAGS=-Wall
# define CFLAGS, CXXFLAGS, LDLIBS and LDFLAGS as necessary
all: name_of_bin
name_of_bin: a.o b.o c.o
clean:
$(RM) -f *.o name_of_bin
.PHONY: clean allPlus I was thinking maybe less magic for someone trying to make sense of it.
Is there no command needed for name_of_bin to be linked? Does it automatically figure it out based on *.o dependencies?
Personally, I would just write "rm -f" explicitly. I don't see any advantages of using $(RM).
Replace your own toolchain for the CC and related commands... really anything that takes multiple files as input and emits one file as output should fit the paradigm.
Which parts are "most of the material" though? ;)
> Replace your own toolchain for the CC and related commands... really anything that takes multiple files as input and emits one file as output should fit the paradigm.
I tried to. It's messy and undebuggable if you run into problems. Especially with tools that already look at the whole project (such as Typescript).
We ended up using Makefiles as just "command launchers" with targets that basically look like
target:
invoke-tool* Clovis L. Tondo, Andrew Nathanson, and Eden Yount (1994). Mastering MAKE: a guide to building programs on DOS, OS/2, and UNIX systems. Prentice Hall.
* Robert Mecklenburg (2004). Managing Projects with GNU Make. Nutshell Handbooks. O'Reilly Media. ISBN 9780596552541.
* John Graham-Cumming (2015). The GNU Make Book. No Starch Press. ISBN 9781593276492.
The only "unusual" things I'm seeing there are the $@ (target) and $^ (dependencies) automatic variables, the wildcard and patsubst functions, which have the description right in the title, and the general ``target: dependencies \n\t command-list`` format of Makefile rules.
And then proceeds to list almost the entirety of the Makefile
cc -Wall -o file1.o -c file1.c
cc -Wall -o file2.o -c file2.c
cc -Wall -o file3.o -c file3.c
cc -Wall -o name_of_bin file1.o file2.o file3.o
rm -f *.o name_of_bin
Disclosure: I wrote it so I apologise for any mistakes
The principle was to create a make target and rule for every host. The rule runs ansible-playbook for this single host only. Running the playbook for e.g. 4 hosts in parallel was as simple as running 'make -j4'. At the end of the make rule, an empty file with the name of the host was created in the current directory - this file was the target of the rule - it prevented running Ansible for the same host again - kind of like Ansible retry file, only better.
I realize that Ansible probably is not the best tool for this kind of job, but this Makefile approach worked very well and was hacked together very quickly.
[1] https://gist.github.com/martinky/819ca4a9678dad554807b68705b...
* Colorize errors
* Hide output unless the command fails
* Automatic help command which shows (non-file) targets
* Automatic clean command which deletes all intermediate files
* Hash-based update detection instead of mtime
* Changes in "Makefile" trigger rebuilds
* Parallel builds by default
* Handling multi-file outputs
* Continuous mode which watches the file system for changes and rebuilds automatically
I know of no build system which provides these features and is still simple and generic. Tup is close, but it fails with LaTeX, because of the circular dependencies (generates and reads aux file).
I've work in embedded software for over a decade, and all projects have used Make.
I have a love-hate relationship with Make. It's powerful and effective at what it does, but its syntax is bad and it lacks good datastructures and some basic functions that are useful when your project reaches several hundred files and multiple outputs. In other words, it does not scale well.
Worth noting that JGC's Gnu Make Standard Library (GMSL) [1] appears to be a solution for some of that, though I haven't applied it to our current project yet.
Everyone ends up adding their own half-broken hacks to work around some of Make's limitations. Most commonly, extracting header file dependency from C files and integrating that into Make's dependency tree.
I've looked at alternative build systems. For blank-slate candidates, tup [2] seemed like the most interesting for doing native dependency extraction and leveraging Lua for its datastructures and functions (though I initially rejected it due the the silliness of its front page.) djb's redo [3] (implemented by apenwarr [4]) looked like another interesting concept, until you realize that punting on Make's macro syntax to the shell means the tool is only doing half the job: having a good language to specify your targets and dependency is actually most of the problem.
Oh, and while I'm around I'll reiterate my biggest gripe with Make: it has two mechanisms to keep "intermediate" files, .INTERMEDIATE and .PRECIOUS. The first does not take wildcard arguments, the second does but it also keeps any half-generated broken artifact if the build is interrupted, which is a great way to break your build. Please can someone better than me add wildcard support to .INTERMEDIATE.
[1] http://gmsl.sourceforge.net
[2] http://gittup.org/tup/ Also its creator, Mike Shal, now works at Mozilla on their build system
: foreach *.proto |> !protoc |> %g.pb.cc | %g.pb.h
: foreach *.pb.cc | *.pb.h |> !cpp |> %g.pb.o
From my point of view it does more than Make with much less. If all I want is a better make, redo is what I use.
Tup is not a make replacement, because it can do strictly less than what make does (e.g. implementing "make install" is impossible because tup only is for building outputs that are local to the project). However, generating correct build files is so much easier because of the guarantees it enforces. This is despite the fact that I after using it my feelings about its syntax have gone from "hate with fury" to "still don't like it, but with the docs open I can figure out the right syntax for what I want"
Most attempts to improve build tools completely replace make rather than adding features. I like the basic simplicity and the syntax, (the tab thing is a bit annoying but easy enough to adapt to).
It'd be interesting to hear everyone's go to build tools.
A script written in whatever the primary language of the project is (usually with some library support), ideally; to reduce the minimum required knowledge to be a first time contributor to the project. Some kind of js build tool for js projects, fake for f#, and so on. I don't want people to need to learn "the one build tool DSL to rule them all" (be that makefile syntax, bazel rules, or whatever) to contribute to a project's infrastructure, on top of the project's primary language.
I've used make quite a bit, and it is doable for individual projects. Where I start having trouble is managing libraries that may be used across multiple projects. When a library needs to supply its own configuration, and be subject to configuration specified of the project using that library, things get rather complicated, which is why I turned to scons.
That's probably in the ballpark, anyways.
The good (and horrible) stuff:
- implicit rules
- target specific variables
- functions
- includes
I find that with implicit rules and includes I can make really sane, 20-25 line makefiles that are not a nightmare to comprehend.
For a serious project of any scope, it's rare to use bare makefiles, though. recursive make, autotools/m4, cmake, etc all rear their beautiful/ugly heads soon enough.
But make is my go-to for a simple example/reproducible/portable test case.
If I didn't have to provide the option to build under Xcode or VS, I wouldn't have to live in the hell that is Cmake.
I'd just use make.
Personally, I'm glad we use cmake at work simply because we have a diversity of preferred build tools: some people like to work in XCode or Eclipse, others like to use a text editor and make, others like to use a text editor and ninja. While we all suffer a bit with cmake, we all benefit from its position as a "metabuild" tool.
In seriousness, the linked paper describes Shake, a Make replacement with two party tricks: one, it is implemented as a DSL embedded in Haskell, thus giving a nice way of programmatic rule generation; and two, it supports monadic dependencies, unlike Make's applicative-only ones.
make clean; make
touchThe most glaring example is .PHONY targets, which are a hack and should just be shell functions. In 'make <foo>', <foo> should be data, not code. 'make test' doesn't make sense, but 'make bin/myprog' does.
I posted this link in another comment showing how Make, Shell, and Awk overlap:
http://www.oilshell.org/blog/2016/11/14.html
Here are some more comments on Make's confused execution model. It's sort of functional/dataflow and sort of not. In practice you end up "stepping through" an imperative program rather than reasoning about inputs and outputs like in a functional program.
https://news.ycombinator.com/item?id=14840696
And at the end of this post, I talk a bit more about the functional model:
So instead of 'make test', I just use './test.sh all', or './test.sh foo'. The test script can invoke make.
The idea is that 'dataflow' parts are done in Make, and the imperative parts are done in shell. This works out fairly well if you're disciplined about it. The only point of Makefiles is to support quick incremental builds. If there's no incrementality, then you might as well use shell. (Note: whenever you use Make, you're using shell by definition. There's no possibility of only using Make. So I take Make out of the picture where it's not necessary.)
For example, all the instructions here are of the form 'foo.sh ACTION':
https://github.com/oilshell/oil/wiki/Contributing
Pretty much every shell script in the repo uses "the argv dispatch pattern"... I've been meaning to write a blog post about that pattern, i.e. using functions with the last line as "$@".
It's when you start having hundreds of sources, targets, external dependencies, flags and special cases that it becomes hard to write sane, understandable Makefiles, which it presumably why people tend to use other systems to generate makefiles.
So sure, understanding what make is, and how it works is probably important, since it'll be around forever. But there are usually N better ways of expressing a build system, nowadays.
```
export GOPATH=$(pwd)
export PATH=$PATH:$GOPATH/bin
go install target/to/build
export GOOS=darwin
export GOARCH=amd64
go install target/to/build
```
which should be simple. Right? Set environment variables, run a command. Set another environment variable, run a command.
45 minutes in and I haven't been able to quite figure it out just yet. I definitely figured out how to write my build.sh files in less than 15 minutes for sure when I started out.
This is why many who use Go don't get all the fuss with makefiles that some folks insist on. You just go to your project directory and 'go test' or 'go build' or whatnot. Simple. No need to 'make test' or 'make build' unless you have some strange, complicated set up.
It uses a private GOPATH inside the repo, created by:
$ cd /path/to/repo
$ mkdir -p .gopath/src/github.com/username
$ ln -s ../../../.. .gopath/src/github.com/username/projectname
It's the best of both worlds, in a way. A Go developer can use `go get` etc. and everything works as he expects. A distro packager can grab a release tarball, unpack it in some random place and do `make && make check && make install DESTDIR=/tmp/build` as she expects.
target:
cd ./dir; ./script.sh cd dir && \
./script.sh && \
./otherstuff.sh
My main complaint about make is that relying on timestamps, while fast, is error-prone and doesn't work with idempotent commands that might refresh the timestamp without altering content.
Now the blog itself is built with make: http://flukus.github.io/building-a-blog-engine.html
1. Syntax higlighting doesn't exist in reader.
2. The index page (http://flukus.github.io/) can't render in reader mode, it didn't hit the "must have a paragraph with at least 68 characters" or whatever the arbitrary limitation is.
For the later I'm hoping the add some sort of meta tag that allows me to enable it in future.
Is there any reason why you didn't include the date of publishing in the pages. Or is it just me who looks for the dates on all the blogs that I read.
It has completely replaced Makefiles for me. It can be used to run shell commands just like make, but the fact that it is written in Python allows you to also run arbitrary Python code straight from the Makefile (Snakefile). So now instead of writing a command-line interface for each of my Python scripts, I can simply import the script in the Snakefile and call a function directly.
Eg.
rule make_plot:
input: data = "{name}.txt"
output: plot = "{name}.png"
run:
import my_package
my_package.plot(input['data'], output['plot'], name = wildcards['name'])
Make, bsdmake and gmake syntax is minimal and precise enough.
https://snakemake.readthedocs.io/en/stable/api_reference/sna...
Admittedly, there are a huge number of arguments to the snakefile constructor, but they are optional named arguments making the constructor safer and easier to use than Java's telescoping constructor and it alternatives (Java Bean pattern or builder patterns). This application apparently has many options or settings.
- make deps to setup/update dependencies
- make serve to start a local server
- make test to run automated tests
- make deploy to package/push to production
- make clean to remove previously built containers/binaries/whatever
There are usually a bunch of other more specific commands or targets (like dynamically defined targets to, say, scale-frontends-5 and other trickery), but this way I can switch to any project and get it running without bothering to lookup the npm/lein/Python incantation du jour.
Having sane, overridable (?=) defaults for environment variables is also great, and makes it very easy to do stuff like FOOBAR=/opt/scratch make serve for one-offs.
Dependency management is a much deeper and broader topic, but the usefulness of Makefiles to act as a living document of how to actually run your stuff (including documenting environment settings and build steps) shouldn't be ignored.
(Edit: mention defaults)
In general also all the implicit it has makes it hard to predict what can happen. Again when you scale to support a project that would be 1) large and 2) wouldn't have a regular structure.
On another smaller scale: doing an incremental build of LLVM is a lot faster with Ninja compared to Make (crake-generated).
Make is great: just don't use it where it is not the best fit.
Good people do not ship software without a way to get rid of it, if needed.
In fact, I realize that the whole idea of using make is probably outright foolish owing to the intertwined nature of the classpath (which expresses runtime dependencies) and compile-time dependencies (which may not be available in compiled form on the classpath) in Java. I'm merely curious if it can be done.
Recently I wrote a similar blog about an alternative app pattern that uses makefiles:
https://zwischenzugs.wordpress.com/2017/08/07/a-non-cloud-se...
pre { font-variant-ligatures: none }If I thought every Makefile had to be like that I'd write ./build.sh too.
target:
invoke-some-tool
Then Makefiles quickly devolve into an incomprehensible undebuggable nightmare with arcane syntax rules and behaviours. It doesn't help that there are next to none good resources on Makefiles.
The next 90% will be to learn that Make breaks when having tabs and spaces in the same file, and your developers all use slightly different editors that will mix them up all the time.
Simple to use without any magic.
https://github.com/quantos-unios/nrmb
Have a look :)
Even though it may not have been originally intended as such, I've found Fabric http://docs.fabfile.org/en/1.13/tutorial.html to be far far more powerful and intuitive as a means of creating CLI's (that you can easily parametrize and test) around common tasks such as building software.
This works in my cases because I have browserify doing all the heavy lifting with respect to dependency management.
I'd say people would lean towards the former, but time and real world experience has shown that sequential dominates everything else.
While things like "npm start" are nice, not all projects are Node.js. In my current startup we're gonna have standardised Makefiles in each project so its easy to build, test, run, install any microservice locally :)
include std.mk
TARGET=foo
HFILES=bar.h
OFILES=main.o foo.o baz.o
include cmd.mk
project(my_software CXX)
add_library(my_lib foo.cpp bar.cpp)
target_include_directories(my_lib PUBLIC include/lib)
add_executable(my_app bluh.cpp)
target_link_libraries(my_app my_lib)
* this can generate ninja files instead of make
* it adds the common targets such as make help, make clean, make install
* dependency graphs can be generated with cmake --graphviz
* solutions for various IDEs
* etc...
You can declare your project structure, metadata, and other build/install/test logic in either. They're thinking about their use cases, not the language and workflow used to achieve them.
edit: and for context, this is usually what happens anyway, since autoconf does it too. it's hard to think of a major project with a manually edited Makefile
For example: using Makefiles to automate static webpage creation and image file conversion.
Still, it probably would be much harder, if possible at all (doubted for most), to achieve the same with any other tool mentioned here.
The truth: go see the configuration of a Jenkins project, and there's a high chance that one of the lines there still says "make".
This is the exact opposite. It supresses echoing the command that is being executed. It's output is still shown like normal.
Shout out to FreePascal/Lazarus yet again!
Make is scary because it's arcane and contains a lot of gotcha rules. I avoided learning Make for a long time. I'm glad I did learn it in the end, though I wouldn't call myself properly fluent in it yet. But there are a ton of gotchas and historical artifacts in Make.