Makeflow: make-inspired engine for large workflows on clusters, clouds, and grids
ccl.cse.nd.edu
ccl.cse.nd.edu
I didn't like make files when I used them in the past.
I get the cleverness of the solution but I don't like the cryptic syntax, something useful hiding behind something (no longer necessarily) complex.
Make is actually a pretty good system for setting up parallel computations, if you're doing it on one machine. Are parallel computations with lots of dependencies too simple of a problem? People get excited about things that are even simpler than that, like Map/Reduce, as long as you're doing them to enough data.
I'm not sure which part you're "not a fan" of -- there are some pretty awful, ugly things you can do with Makefiles, which includes basically everything about compiling C, but you don't have to do awful things, and the fundamental idea of "you need this to build this" is implemented very well by Make.
Now, I've recently been converted to using Ninja, in fact by someone on HN, when I asked if there was a Make-like system that understands steps with M inputs and N outputs. It keeps the same fundamental idea while discarding a lot of Make baggage. But if this Makeflow system said "it's like Ninja for clusters" instead of "it's like Make for clusters", nobody would know what they were talking about.
This is not to say that there's a problem with GNU parallel, it just seems like it's designed for the "embarrassingly parallel" case with no dependencies between steps, whereas Make and the system under discussion are for the case where you have dependencies between steps.
I've also never found myself needing to use xargs in a Make or Ninja build, incidentally.
This isn't "poor man's MapReduce" anymore than a car is a poor man's space shuttle. A space shuttle is a difficult and expensive way to get around if you don't need to leave the surface of the Earth. A MapReduce framework is a computationally expensive way to get data around if it doesn't need to leave your computer.
Look at Ninja sometime. It's not a hack.
Although make can be used to parallelize data workflows (guilty as charged), it was not particularly designed for this use. It has lots of other things baked in that specifically target building software which for the purpose of data work flows are extra cruft. Effective ? yes, quirky? yes, recommended ? perhaps not if one is not very familiar with make.
Ninja is lovely its like a library of building blocks to make your own build system. BTW as long as we are trading favorites, may I suggest tup.