Doit: bringing the power of build-tools to execute any kind of task

Dolt!?

( http://www.folklore.org/StoryView.py?story=Do_It.txt )

I was looking for a build-tool like system to "compile" my thesis: run data analysis scripts, output figures, do some munging on latex source and bibtex references, and finally run the latex compile.

After trying a few and fighting with waf when I needed more flexibility, I found doit to really be a godsend. It really seems to be the best mix of simplicity, flexibility, available features and extendability. Thought some of you might also find it useful. Plus, no arcane make syntax to learn or remember.

I have used PyDoit for various automation tasks and found it pretty useful. The author is very responsive as well and fixes problems in a timely manner. It's a good alternatice to Makefiles and libraries like fabric. Parametrization is an issue of course, but tasks can be easily customized using generators, which adds a lot of flexibility.

What are the benefits compared to make in a unix environment?

I have been using snakemake, which is a similar tool, but really nice, and specifically designed for scientific workflows. https://bitbucket.org/johanneskoester/snakemake/wiki.

I made this to achieve more or less the same thing: https://github.com/crdoconnor/projectkey

I focused more on "saving keystrokes", though. Hence the autocomplete and the use of a one letter execute command.

I've looked at a couple of data build tools and many will do basic jobs very well, and some of them will use even a nice dsl. Where I see many tools struggle is with task parameterization, which I found to be a frequent requirement for these kind of systems.