Hacker News Reader

Comment by majewsky | Hacker News Reader

> Using make in a CI system doesn't really work, because of the way it handles conditional building based on mtime.

Any CI I've seen starts from a fresh repo checkout and rebuilds everything every time, so it's not an issue with practice.

OTOH, probably all my projects were small enough for it to not hurt when the CI builds everything from scratch every time. I might look at this from another angle if the things I worked on were mega-LOC C++ programs, and not kilo-LOC Go programs.

View on HN

That works, until your build system is sufficiently large or time consuming or not C/C++ like make was originally built for. For example, at my company we have a build system for building all of our Amazon Machine Images (AMIs). It doesn't make sense to re-build images unless they're dependencies have changed, but mtime just doesn't make any sense for a build system like this. Trying to coerce make into doing what we wanted was like pulling teeth.

Why didn't you just take make sources and fix them for <whatever> instead of mtime() check, making it a command line option or environment variable? Or simply .MTIME target, which would return int-like string of rank, defaulting to mtime() if not declared.

I guess it's because of his point #3.

Honestly, if I were to modify make for some custom behavior, I would look for another tool¹ or create yet another builder tool. Because although I already know most of the syntax, point #2 is an incredibly big problem, the GP missed point #4 in that make does not manage external dependencies², and the syntax is not bad only for learning, but for using too.

1 - Almost everybody seems to agree with me on this. The only problem is that we have all migrated to language constrained tools. This is a problem worth fixing.

2 - Wether it is installing them or simply testing if they are there, make can't do any. Hello autotools, with it syntax set to extreme cumbersomeness.

I don't know if you really understand Make. Even the original article misses point about what Make is really about in recommending `.PHONY` targets.

Make is about "making files." Or to be a little more semantically specific, Make is about "processes that transform files to make new files based upon their dependencies". It really doesn't matter that your file is a C source file, or some unpreprocessed CSS, or a template, or an AMI. To your AMI example, if you specify a dependency (or dependency chain, DAG) as a time-stamped (or set of) file, you can get make to rebuild the AME for you along with any other supporting or intermediate files.

IMO, the suckless guys are masters at writing deceptively complex but highly readable & concise Makefiles. Here's plan9 workalike utility library [0], a util-linux/busybox package of binaries [1], a window manager [2], a terminal emulator [3], and webkit2 based web browser [4]. I highly recommend you study them if you're looking to up your Make game.

[0] http://git.suckless.org/9base/tree/Makefile

[1] http://git.suckless.org/sbase/tree/Makefile

[2] http://git.suckless.org/dwm/tree/Makefile

[3] http://git.suckless.org/st/tree/Makefile

[4] http://git.suckless.org/surf/tree/Makefile

The first example uses recursive make, breaking the graph :(.

I'm not saying you can't use make (we did use make before we switched to walk), it's just more painful for non C/C++ build systems. All we really want from a build systems, is a generic tool for describing, and walking, a graph of dependencies.

Recursion does not imply a circular dependency which is most people's biggest concern with Make recursion. A graph with loops is still a graph. A broken graph is actually 2 graphs.

If you're careful (and even if you're not), loops in your dependency graph are usually a non-issue. And if you use a single `Makefile` it'll detect the circular dependency and try to ignore it.

Let's take an example processing Sass CSS files. You just need 2 folders in your project, `css/` and `scss/`, and a `Makefile` to process all your Sass files into CSS.

   # Locate our source assets, save a list
   SRC = $(shell ls scss/*.scss 2>/dev/null)
   # Specify a filename transformation from scss to css and convert the SRC list
   OUT = $(SRC:.scss=.css)
   # Define rule of how a scss to css transformation is supposed to happen
   css/%.css : scss/%.scss
        sass $<:$@
   # Make the `all` target depend on the OUT list
   all : $(OUT)

It takes just 5 lines to teach `make` how process Sass files. This would process any `.scss` file you dropped in `scss/` and save it in `css/`.

Sorry but that makefile is repeating the absolutely most common mistake in make.

If you use @import in your scss file the css will not be rebuilt if only the imported file is modified.

So no. It does not take 5 lines to teach make how to build Sass. To solve this you either need to teach make about sass @import, or teach sass about make and let it generate makefiles (like gcc -MMd). Or simply just use sass --watch for incremental builds and take the recompile hit if you need to restart it for whatever reason.

(While I'm at it...Another thing missing from that makefile is source maps (.css.map) generated by the sass compiler. It's not only one css file being generated from each sass file. That will complicate the rules even further)

Yet another error is that if you remove one scss file, the old generated css file will not be removed after a rebuild. If the next build step does something like wildcard-include all *.css-files you will get problems.

Blind includes are a dumb to do (it's one easy way to exploit a source codebase). The better way is to generate your include from, you know, the actual way your codebase is. Use `$(OUT)` to build your include statement. That var says, "load any of my files", not the lazy/dangerous form of, "load any files that might actually be there".

And yes, by that logic, `SRC = $(shell ls scss/*.scss)` is dumb. That is not lost upon me.

You don't need to (specifically) teach make about `@import`. It's another scss file and it (hopefully) should be a part of your SRC list. Yes, having something like `gcc -MMd` would be better but that's sass for you.

To fix the `.css.map` issue is simple. You can't use the pattern you've seen for yacc/bison, though. The `$@` var would have both files in it.

   %.tab.c %.tab.h: %.y
           bison -d $<

Instead, simply adjust the pattern to rule to be the functional equivalent:

   css/%.css : scss/%.scss
        sass $<:$@
   css/%.css.map : scss/%.scss
        @touch $@

Even if the imported file is part of SRC list, any files importing it will not be rebuilt if only that file is modified.

Ahh. Got you. And easy fix again. Add `$(SRC)` to the pattern rule.

   css/%.css : $(SRC) scss/%.scss
        sass $<:$@
   css/%.css.map : $(SRC) scss/%.scss
        @touch $@

Now you will get correct results. But now if you change just one file, you will always rebuild everything, even files that don't import the changed file. Maybe not a big deal for SASS as you usually don't have mega LOCs of SASS and compiling is fairly quick. Just want to highlight that make isn't always as simple as it looks at second glance.

> To solve this you either need to....

Or teach a third-party tool to output a list of @imported files for a given Sass file. Then this list can be used as that Sass file's dependencies.

Most languages don't build as quickly as go. Having to wait tens of minutes for a clean build is not unusual.

That brings back not-so-fond memories of trying to compile Servo on my notebook. :)

By the way, I just noticed that compilation times are another argument for microservices.

CircleCI (at least) caches build dependencies for speed, but it does it based on specific directories, not timestamps. As a result, it's not as fast as it could be because unless you munge your build to fit, it doesn't cache intermediate build products of your own code.

> so it's not an issue with practice.

I meant "in practice", of course...

Reply on news.ycombinator.com