Comment by emelski | Hacker News Reader

Comment by emelski | Hacker News Reader

If you're looking to serialize parallel build logs without changing to an entirely new build tool:

1. Electric Make, a high-performance reimplementation of GNU make and ninja, has properly serialized build output logs since it was introduced in 2002 (https://electric-cloud.com/products/electricaccelerator) (disclaimer: I'm the chief architect for ElectricAccelerator, of which Electric Make is a component).

2. I published a technique on CM Crossroads you could use with GNU make 3.81 to descramble parallel build logs in 2009. The article has moved around since then but these days it seems to be found at https://www.cmcrossroads.com/article/descrambling-parallel-b....

3. The maintainers of GNU make took the concept described in that article and baked it into GNU make itself in 2013 for version 4.0 (http://git.savannah.gnu.org/cgit/make.git/tree/NEWS?h=4.0)

That is a good article, but it doesn’t seem to address several of the concerns raised in my redo article. In particular, it looks like the final output is still in a nondeterministic order, it doesn’t print output from a given target incrementally while it runs, and you can’t query the log retroactively to look for clues in a very big run.

It’s a big advantage to not have to change tools, of course. Although redo happily interoperates with makefiles (which is how the buildroot patch works) so there’s no need to convert everything to get most of the advantages.

Correct, both the technique described in that article and the feature that eventually wound up in GNU make only disentangle output from concurrently executing build processes. With significantly more work GNU make could probably be made to enforce a deterministic order.

However, Electric Make does emit the output in a deterministic order, in fact in exactly the same order as the build would have produced had it run serially. In practice the delayed output is not such a big deal -- people just don't really seem to care that much, when the build overall finishes 20x - 30x faster than it used to. Electric Make also generates an annotated build log, essentially an XML-marked-up version of the log, which contains a tremendous amount of additional information about the build and vastly simplifies debugging, actually.

Hey, Electric Make looks really cool - I ran into it while I was tinkering with https://github.com/dan-v/rattlesnakeos-stack. I also have done a fair few builds in buildroot. It wasn't clear to me at all if you offer anything for open source work / hobbyists - I'd love to try it out.

Reply on news.ycombinator.com