One of the important paradigms that I think Luigi and Airflow miss is that they treat pipelines as a DAG of tasks, when it really should be thought of as a DAG of Data.

It's a subtle difference, but has huge impacts when you're trying to dynamically scale tasks based on cluster resources and track data lineage throughout your system.(Disclosure: I'm the founder of Pachyderm[0], a containerized data pipeline framework where we version control data in this way).

Check out Samuel Lampa's post[1] about dynamically scaling data pipelines for more details.

[0] github.com/pachyderm/pachyderm [1] http://bionics.it/posts/dynamic-workflow-scheduling