Comment by sorbits | Hacker News Reader

Comment by sorbits | Hacker News Reader

Parent Full thread

> As a shameless plug, I built a tool similar to Make and redo […] https://github.com/ejholmes/walk

Your README’s example show `parse.h` as output from `Walkfile deps parse.o`, I think that is a mistake.

As for your build system (and comment), I have some questions:

1. How do you achieve using a deterministic hash as condition (and aren’t all hash functions deterministic)?

2. Why would you not be able to use mtime as a dependency? The only case I have run into is when the build depend on data from remote machines, but in that case, I think the proper solution is to have an initial “configure” step where you retrieve the data your build depend on and/or a build rule to update this data.

3. Does your build system execute the Walkfile for every node in the graph on each walk? Because that sounds like a quite noticeable overhead for larger projects.

4. Am I right in thinking that the primary advantage with your system, over make, is that a shell command is executed to obtain a target’s dependencies?

1. Yeah, hash functions are deterministic, but your input needs to be determinsitic across machines too. For example, on a unix system, you may want to conditionally build if any files have changed. To do that, you could generate a deterministic hash of the dependencies with something like `find . | sort | xargs sha1sum | sha1sum | cut -d ' ' -f 1`. Including mtime in that would break across machines.

2. Mainly because doesn't translate across machines; it only applies to your machine. If someone checks out the repo on their machine, mtime is different. As soon as you move a build system to CI, you need something better than mtime, like content adressable hashes, if you intend to cash targets.

3. It executes the Walkfile twice for each target in the graph; once to determine the targets deps, and once to execute the target. This definitely hasn't been a bottle neck for anything I've used walk(1) with so far.

4. Correct! But even more so, replace "shell script" with "executable". The Walkfile can be written in any language you want, as long as it has the executable bit set.

Thanks for the clarifications.

As for #1, what I do not understand is how a `Walkfile` allows me to use a content hash change to trigger a rebuild.

Your documentation says that a file list should be returned for `deps`, so how does the `Walkfile` communicate that e.g. `main.o` should be updated if `sha1sum main.c` is different than on last invocation?

Good question. It's not a concern of walk(1) itself, since it's impossible to generalize for every use case. walk(1) _always_ executes the target, but the target itself can decide whether it actually needs to do work. For a C project, you would just do something like this: https://github.com/ejholmes/walk/blob/master/test/111-compil....

That's a trivial example using mtime. You could replace it with a hash function if you wanted.

> your input needs to be determinsitic across machines too

This is one of those things that "distcc" and "ccache" have dealt with effectively - anyone trying to build big C++ projects would be well served by looking at those tools.

Reply on news.ycombinator.com