Why Use Make? (2013)
bost.ocks.org
bost.ocks.org
Most of the common criticisms of Make are actually criticisms of autoconf, which I agree is a hideous tool. (On the other hand, autoconf is a tool intended to address a hideous problem, so perhaps that's inevitable).
A lot of the more recent build tools I see are largely reinventions of Make. Make is very widely supported (most basic projects can make do[0] with portable Makefiles, though GNU Make is also available for most systems), and its syntax is actually very easy to grasp and manipulate[1].
[0] no pun intended
[1] Most projects only need a very small subset of what Make has to offer, anyway, and that can be learned in a matter of minutes.
[1] Something like http://mad-scientist.net/make/autodep.html
Incidentally, I am working on a make-replacement that is intended to address this and similar needs. It's still in very early stages, but I think I'm onto something.
The idea is to write a very low-level tool that leaves out most of Make's higher-level abstractions. In my tool there are no recipes, no implicit rules, no variables or variable substitutions, no conditionals, etc. The input to my tool is just the precise specifications of the commands you need to run, their inputs and outputs (so a precise dependency graph can be calculated), and with everything fully-expanded already.
The idea, then, is that whatever higher-level abstractions you want (if any) you build into a higher-level tool. The higher-level tool just spits out a file describing the list of tasks. Then the higher-level tool can worry about the higher-level structure, policy, configuration, etc. of your project. So instead of writing things like implicit rules in Makefiles, you just write some code that explicitly generates tasks.
For example, with Make, you might write an implicit rule like this:
%.o : %.c
$(CC) -c $(CFLAGS) $(CPPFLAGS) $< -o $@
for c_file, o_file in files:
tasks.append(Task(
target=o_file,
source=c_file,
command="gcc -c %s -o %s", c_file, o_file
))
print(tasks)
I have an elegant solution for the auto-dependency problem (unproven, but I think it's promising). The idea is that your dependency-calculating tasks are still just tasks, but the tool knows how to integrate the calculated dependencies back into the overall dependency graph. These dependency-generating tasks depend on the files they are generating dependencies for, so it is all part of the unified dependency graph.
If you're interested, star my project and follow my blog (where I will make any announcements about it):
def outname(f):
return f[:-1] + "o"
@file_rule("*.c", outname)
def compile(f):
#do the compiling here
I agree with you that Make's pattern matching facilities are ugly and could be improved upon. I tried something of the sort [3] with redo, and it worked well, but felt too alien.
From what I've seen at big companies with huge builds, the "dependency graph itself changed, now what do we rebuild" problem is the killer problem. I bet you've seen instances of that at Google.
So, to compile a slightly complicated project, I would need Ruby for the main project, Python for a library it builds on, Haskell for another library it builds on, etc?
In theory, that's not a new problem, but I think that, by removing the common language that make provides, it will become much more common in practice.
At the extreme, it would only be natural, but disastrous, for the guys writing language X to use language X to write their make file. Result: you need to bootstrap language X through a zillion versions to port it to a new system, or mess around trying to build its parts by hand.
return ctx(
rule = 'candle.exe -nologo -out ${TGT} ${SRC} -dMySource="%s"' % dirpath,
source = ['%s.wxs' % group],
target = ['%s.wxsobj' % group]
)
ctx.objects(source = glob.glob('src/*.cpp'), target = 'OBJS')
Whilst salt is not aiming at Make-for-DevOps it certainly is something that is in the realm of possible.
The gap between compiling a bunch of source code that becomes an executable and a bunch of source code that becomes twenty servers running fifty apps is very very small.
> Ninja is a small build system with a focus on speed. It differs from other build systems in two major respects: it is designed to have its input files generated by a higher-level build system, and it is designed to run builds as fast as possible.
Obligatory: https://xkcd.com/927/
Also played a bit with tup, and it's quite nice too.
Yes, I came across Ninja when searching for prior art. While it's "philosophy" essays make it sound like it is exactly what I'm looking for, when I dug in deeper I found that it was still a ways away from what I had in mind.
When you look at Ninja files, you find some of the things that I explicitly want to get rid of: variables and rules, evaluation and scoping. Their solution for C/C++ dependencies is to have special magic in the tool.
Maybe the biggest difference between Ninja and what I am envisioning is that, although pared down and low-level, Ninja is still a command-line tool implemented as a language (.ninja files). What I have in mind is an embeddable library that deals only in flat, fully-expanded lists of tasks. At its core it is much more like an API than a tool (though wrapping it in a convenient command-line tool will of course be an important use case).
Think about it this way. There are so many directions you might want to extend your build system:
1. you might want to run your jobs in a different way, for example by distributing them to a cluster of build workers.
2. you might want to run your builder as a daemon that watches the filesystem and receives notifications of changed files, instead of having to figure everything out from scratch every time.
3. You might have custom, per-language logic for calculating dynamic dependencies.
4. You might want to fingerprint your inputs and outputs and cache artifacts so that you can return cached results instead of doing redundant work (especially useful if you have a build cluster that many people share).
5. You might want to do strongly isolated and hermetic builds that run each command against a directory tree that only contains its supposed inputs.
6. you might want to closely integrate the build system with your IDE, so the IDE can show deep and up-to-date information about the progress of the build.
7. I think of Light Table-like environments as being the future of IDEs. Part of the philosophy of Light Table is that you can program your own custom editor interactions easily ("The future is specific": http://www.chris-granger.com/2012/05/21/the-future-is-specif...). Programming a specific editing mode would often involve calculating artifacts derived from your source files to show in a different editor pane (for example, the assembly corresponding to your C, or a graphical representation of your source). To do this, you want something like "make", but programmable so you can generate and remove rules on the fly.
With a Ninja-like approach, you would have to add these features to Ninja itself. But once you do this, Ninja is going to get even less small and simple.
This is why I think the way forward is to have an API-based, embeddable builder engine. The core just knows how to take fully-expanded task specifications and evaluate the dependency graph to decide what jobs need to run. Then the logic for how to run jobs, record their results, possibly return cached results etc. can be easily mixed and matched. For example:
1. I would ship with a very simple command-line tool that acts mostly like "make" or "ninja": you run it, it looks at the filesystem to decide what to do, and then does it. When the build is finished the tool exits.
2. I would likely also ship with a daemon version of the same that watches the filesystem with inotify/dnotify and automatically keeps outputs up to date when inputs change. The daemon would likely support some kind of service API that you could use to programmatically query the list of tasks and jobs, and probably also an HTTP/HTML UI.
3. Anyone who wanted to integrate with a distributed builder service could embed the library but write completely custom code for actually executing the jobs.
But here is the key thing: the Tasks that a project writes can stay the same no matter what fancy stuff gets built on top. Whether you are doing the dead-simplest local build from scratch or whether you have a long-lived daemon that distributes your build and heavily caches your artifacts, there is still a common data model for tasks and jobs, and still a common dependency-evaluating core that can be shared no matter where the library is embedded.
You can do something like the make variant with:
: foreach *.c |> gcc -c %f -o %o |> %B.o
: foreach *.c |> gcc -Wall -c %f -o %o |> %B.o
: *.o |> gcc %f -o %o |> hello
run ./myscript.py
http://www.reddit.com/r/programming/comments/21i19h/a_generi...
until you get to update a codebase were all the build scripts are the worst form of perl/forth/ksh/awk scripts all written by someone who though a new script on top of the old one would be easier
I think the main issue with approaches like yours is that there's no way to verify correctness, so it's easy to introduce bugs.
- I'm just not a big fan of automating the configuration of a build system - you build/put yet another machine around a machine. As long as it works - ok, but as soon as you run into unexpected behavior it will become a huge time sink to debug. When you update the Makefile contents manually you take ownership over everything and you get a deeper understanding on how everything works together, right down to the command parameters sent to compiler and linker. If you don't want to do that I'd suggest switching to something like cmake - if it has to 'magic', at least have the behavior specified in a coherent manner.
Not at all sure what autoconf has to do with anything, autodeps are a completely separate problem space.
http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1
which points you to this:
https://www.freebsd.org/doc/en/books/pmake/
But really, if you just hate using info(1), always remember that GNU also puts the pages on the web in HTML multichunks, HTML one big page, and PDF, so there's no reason to dodge a GNU tool just because you hate info(1).
info --subnodes "$@" 2>&1 | less
Envoy pynotify and fnmatch pretty much do the heavy lifting.
I do it this way because
Since I know Python no extra syntax knowledge is required
Python is going to be readable to me in 6 months when I've
forgotten the syntax of whatever build tool I just used
*All* my machines desktop and servers already have Python
installed
*It's fast, merging 20-30 largish text files takes around
10-15milliseconds
* It's very very flexible (it's Python)
It's an incredibly powerful and clever tool hiding behind an interface (that second to Git) is the worst I've ever had to use.
EDIT: I've actually been toying with the idea of writing a python library to simplify things as lots of the code I end up writing is very similar across projects, it's been one of my "think about in shower" projects for quite a while.
That's basically scons. Give it a try, I've been using it for years to manage a fairly complex, unusual build process with minimal effort (less than 200 lines of Python code).
Scons is awesome but for what I use Python for (minification of web assets, automatically running image optimisations) it's taking a sledgehammer to a walnut.
Now, this approach depends of course how well you know make to begin with. But there are a lot of examples out there, and the man pages of GNU make are quite good. It is quite true to the old make facility, so even older book examples should work with it.
The reason I roll my own in Python is that I get exactly the functionality I want working in exactly the way I want and quite often I can replace a binary dependency which I use a tiny part of with a python function.
I find that reduces my cognitive load as I'm only working with one thing instead of others.
ymmv.
If that has been your only exposure to Make, you should take another look at it, and consider using it for your next small- or medium-sized project.
This isn't to say that I think Make hung the moon in radiant perfection, but I haven't seen any tool that has a clearly superior set of compromises and tradeoffs. Gorgeous example: automatic dependency generation. Does anyone really think the identical dep generation codebase will work on 20 year old C and node? It's not hard to glom whatever dependency mapping you want into your make workflow and drop it into separate included makefiles.
It feels easier to write your own. But what that really leads to is re-learning all the shit the make maintainers have learned in the last (look it up.. DAMN.) nearly 40 years.
Or maybe _not_ learning it, and duplicating mistakes which have been solved for decades.
For example: If I'm working on a C++ project then I'd like my build tool to be aware of concepts like header files, shared libraries, include paths, compiler options. For me that tool is CMake.
Can I do that in Make? I'm sure I could, but I'd end up with the same problem you're complaining about. I'd essentially be writing my own build system, just implemented in Make. Life is too short for that.
Add to that diamond shaped dependencies such as several h files including another generated h file and you're in for many sleepless nights.
Almost every problem with makefiles is nothing to do with the tool itself - it's something to do with the environment, and the tool not being sufficiently intelligent enough (read: psychic) to figure stuff out. Most of the hacks built on top of make (eg, pkg-config) are attempts to fix some of these problems, but IMO, they're going about it from completely the wrong direction.
The Nix/Guix approach - declare your entire environment up front - will greatly simplify the requirements of any build system - there no longer any magic involved. You won't necessarily make the same mistakes, or even need to consider half of the problems Make has had to deal with over the years.
Make is still relevant and useful in combination with Guix/Nix, but it shouldn't need the hacks built on top of it, like pkg-config.
I cannot take this seriously.
I like the concept of make but it doesn't make up for its own warts. Unfortunately, there is no single build tool I can blindly recommend to people without being extremely familiar with their project and how it builds. No solid and lightweight build tool that pleases more or less everyone without having 183 pages worth of manual and a repugnant syntax.
CMake/Lua gives me hope but we're not quite there yet. Make it pretty decent for small projects though... I see it as the HTTP of build tools, though. It has serious issues but when tools come up they are built on top of make because it's ubiquitous.
For example, here's how I handle a simple C project.
CSRC := [list .c files here]
DEPS := $(CSRC:.c=.d)
[standard statements for building go here]
%.d: %.c
gcc -MM -MF $@ $<
-include $(DEPS)
1. dash before include. It will cause make to ignore any errors when generating the .d files. It can be pretty difficult to figure out what went wrong. Removing the dash will show a (harmless) error message for every .d file generated, which is pretty annoying.
2. At one time a.c includes b.h and you run make. a.d is created with "a.o: a.c b.h". If at this point you add to b.h an include to c.h, the new dependency of a.o on c.h will never be updated in a.d unless you manually delete it. This can cause some pretty nasty bugs! This can be solved by adding
-MT $*.o -MT $*.d
tl;dr: this solution leaves much to be desired, and can be dangerous in some conditions.
CFLAGS+=-MMD
-include $(DEPS)
It doesn't require differentiating between identifically rendered tabs and spaces. It doesn't need dependency hand-holding, with explicit touch or checking Last-Modified. The simplest script just builds it all from scratch, like make clean.
But it sure isn't that much more complex nor requires a lot more of your system, so I'd say it's more a matter of preference.
To write your workflow purely in shell scripts, you will invariably use lots of tests.
Also note that they are not necessarily mutually exclusive. You can write Makefiles that call shell scripts (and shell scripts that call make)
SHELL=/bin/bash
at the top of the Makefile when I use bashisms to construct complex pipelines.
And once, in a fit of mad genius, I made a particularly complex and useful Makefile executable by putting #!/usr/bin/make -f at the top. (It was a bad idea, obviously, and I won't be doing that again.)
For large file based datasets, the shell is just a REPL.
- straightforward and obvious support for multiple named entry points
- by default, process terminates when a command returns a non-0 exit code
- often-adequate (though still crappy, especially for paths with spaces) set of built-in string processing functions
- scripts often somewhat portable between Unix-style shells and Windows
(I suppose there may be Unix-style shells available for Windows, but I've never found a low-dependency one that actually works. So combined with the first three advantages I've stuck with make.)
I've actually found Make much better as a shell script replacement than it is for its stated purpose of building software :)
Imagine that you needed to download a tar archive, unpack it, then run several simulations followed by regressions followed by figure plotting on the data. You could write a shell script to do this, but it would be hard to make the shell script simulate the capabilities of `make -j', and you'd have to do a lot of timestamping and file existence checking to simulate the incremental computation capabilities of make.
I worked on a proprietary build system remarkably similar to gradle for many years. The cleverer a build system tries to be the more likely it will become a self sustaining beast that consumes ever more of your time and destroys productivity. They look great with relatively simple systems, but a few years into production with multiple deployment target configurations and you will want to murder people. Build systems should be stupid, simple, and trivially predictable.
I know Gradle fairly well, and i wouldn't describe it as particularly clever. The basic ideas in it are quite simple (and actually fairly similar to make!). Most of the complexity comes from specific build tasks for specific purposes, which doesn't complicate the core.
So it is rather straight forward. Type make, sometimes autoconf and most of the time you are done.
Also in my own projects I prefer make to many other build tools. In software development, command line still rules (if you want to be really productive)!
There's some shell script to glue the pieces together, and a Python script for invoking processes within the ERP applications, but make is the secret sauce that figures out what order to run everything in.
It's working pretty well so far. It wasn't an intensional thing. I had a fairly complex stack for a particular project that I needed to share.
https://news.ycombinator.com/item?id=7487202
exactly because I like the idea of capturing the setup (compared to a more haphazard approach). My initial reaction was that for the computer setup use case, it did too much to hide away complexity.
I guess I don't have a deeper point, but I wonder if it is overkill for capturing simple workflow like is discussed here.
http://blog.factual.com/introducing-drake-a-kind-of-make-for...
Do you know of Unix pipes? The fact that gulp calls all of this "streams" and "pipes" is a seriously bizarre form of NIH.
You might also be interested in make -j option.
(sarcasm)
This is yet another example of the JavaScript community being completely ignorant of what came before them.
If there was a version of make I could point to that would work easily on windows, preferably as a non-global variable, I would use that, until then I probably will have to use something else, through likely just bash scripting with pipes (which does work in windows).
The one here:
http://gnuwin32.sourceforge.net/packages/make.htm
works fine if you do set path= and then call it using a full path into Program Files (I've just checked to make sure, but I used whatever version I had installed, not sure it's the latest one there).
If you download any complex project, there will almost certainly be a meta makefile generator (autoconf, cmake, etc) or a set of about 30 makefiles for different various specific situations (lua, etc).
...because make is terrible at doing complex tasks on its own, and terrible at code reuse on it's own.
To be fair; async I/O isn't a magical button that makes gulp better than anything else, but it's a bit off saying an entire community is ignorant. Don't be a dick.
The important points about gulp are:
- Better syntax
- Plugin system (code reuse)
- Functional programming style
- Faster than its main alternative, grunt
http://aegis.sourceforge.net/auug97.pdf is old, but the problem it documents is still fairly common.