Redo, buildroot, and serializing parallel logs
apenwarr.ca
apenwarr.ca
1. Electric Make, a high-performance reimplementation of GNU make and ninja, has properly serialized build output logs since it was introduced in 2002 (https://electric-cloud.com/products/electricaccelerator) (disclaimer: I'm the chief architect for ElectricAccelerator, of which Electric Make is a component).
2. I published a technique on CM Crossroads you could use with GNU make 3.81 to descramble parallel build logs in 2009. The article has moved around since then but these days it seems to be found at https://www.cmcrossroads.com/article/descrambling-parallel-b....
3. The maintainers of GNU make took the concept described in that article and baked it into GNU make itself in 2013 for version 4.0 (http://git.savannah.gnu.org/cgit/make.git/tree/NEWS?h=4.0)
It’s a big advantage to not have to change tools, of course. Although redo happily interoperates with makefiles (which is how the buildroot patch works) so there’s no need to convert everything to get most of the advantages.
However, Electric Make does emit the output in a deterministic order, in fact in exactly the same order as the build would have produced had it run serially. In practice the delayed output is not such a big deal -- people just don't really seem to care that much, when the build overall finishes 20x - 30x faster than it used to. Electric Make also generates an annotated build log, essentially an XML-marked-up version of the log, which contains a tremendous amount of additional information about the build and vastly simplifies debugging, actually.
Not at all! In fact there are other tools in other domains that could benefit from careful thought about how to interleave parallel outputs in a way that favors interactivity/immediate output but preserves (effective) ordering [1][2]. This is a tough problem a lot of tools just give up on, but the status quo of "buffer everything until it is completely done" grates against one's senses.
> I'm actually curious about things like ansible; what exactly does it do that you couldn't just do with a bunch of redo scripts?
That might be more of a philosophical question :-) Most of those tools could be replaced with a sufficiently intelligent set of scripts. I think they get their popularity from providing more convenient syntax, building in parallelism for applying changes across hosts, offering cross-platform ways of doing common admin actions like creating accounts, etc.
But if you look at them the right way, they definitely share a lot in common with build systems, particularly when it comes to managing dependent actions.
* https://news.ycombinator.com/item?id=14837740
* https://news.ycombinator.com/item?id=14928216
* https://news.ycombinator.com/item?id=14836340
* gopher://jdebp.info/h/Softwares/nosh/guide/external-formats.html
* http://jdebp.eu./Softwares/nosh/guide/external-formats.html