Hacker News Reader

Comment by jonhohle | Hacker News Reader

Tracking dependencies is what makes make make. Love it or hate it, Ant got this right (imho), and modern build systems almost universally get this wrong. Every language can run and compose tasks. Dependency tracking, parallelization, and language agnosticism are the killer features of make (again, imho).

View on HN

I agree with you on all grounds (I personally use make daily) however note that `make` checks dependencies via last updated time flag on files. I think it would have been a better default to cache based on hash of file. This way if document was modified, by hash wasn't changed, make shouldn't rebuild tasks that depend on this file.

But computing the hashes of all the files would take the same time as rebuilding everything, or maybe even slower. And you'd need to re-compute all these hashes even for a no-op. It's absurd.

The nice thing about dates is that they can be checked instantaneously, and independently of file size. If a file has changed, its date has, so it's all the information you need.

Computing the SHA sum of a file is plenty fast on modern hardware, nowhere close to the same time as a rebuild.

Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command:

time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2 | sha256sum

Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB.

For raw throughput, you can use BLAKE2 for even more speed https://www.blake2.net/

What % of that time is required for just computing the timestamps of these files? I bet it's a few orders of magnitude faster.

Sure, but who cares?

While 1280 files isn’t exactly small, it’s not really big either. Big as in where incremental builds really show their shine.

This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat.

The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.

Rebuilding potentially has side-effects. You can have a build system that, if a file is changed, it makes HTTP calls, generate docker containers, store artifacts in S3 etc. Given this, computing the hash would be negligable amount of work. Besides make does iterate over all dependencies every time to check last modified date filesystem keeps. So, at that point one can compute hash.

You shouldn’t have such a build system. Idempotence is a feature, not a bug.

idempotent does not mean side effects free; a compile-to-the-cloud system for distributed caching can be idempotent (ocaml is building support in this direction)

Side effects that target on non local dir seems shouldn't be included in the build step. It should be in a standalone script/ target/whatever. Or the build process will not be reproducible at all.

builds should not use non-local inputs, but they can have non-local outputs. the simplest case would be automatically publishing a package or image to a private registry

> If a file has changed, its date has, so it's all the information you need.

Unless the date hasn't changed -- there is no hard guarantee that mtime will be accurate.

If the file's contents have changed but the timestamp didn't, then something is broken and _that_ is the thing to fix, not rewriting the fundamental mechanism underneath the oldest relevant build tool.

There are always different preferences on deciding when something needs to be run again. File modification time is a reasonable default that serves many (most?) cases pretty well.

Yes, file hashes might be another way. On the other hand, if I modify a comment, the file hash will change, but re-compilation is not necessary. In this case, you might need yet another way...

You can always write your Makefiles to compute and depend on hashes or whatever else you might consider appropriate for your exact use case.

I've been using make for a long time but make really really sucks at parallelization and language agnosticism.

I mean those things work, but they're by no means nice. Parallelization works, but output of different tasks (and even the log that certain commands have been run) are interleaved, making it impossible to view progress. Language agnosticism works by operating on the level of shell commands, but that's really about it. If shell commands need to be customized on a per-system or per-OS level... well it's possible to hack around that with enough external scripts and even esoteric if statements inside the Makefile, but it's not nice.

Documentation for make feels like it's from the 80s. Which it is.

All these things (with the exception of having to rely on arcane sets of external scripts and if-statements-in-Makefile) are perfectly fixable. But they don't get fixed.

I haven't understood what the point is for a "task runner" over having a bin/ directory in the project with some shell scripts. If it's not going to do anything extra for you (like run commands according to a dependency tree like make), I don't see the point.

Reply on news.ycombinator.com