Toil – A Python Workflow Engine

Think of it more like an "execution engine" rather than a workflow engine. You define multiple jobs, their resources, and how they interact. The engine executes each step.

Unfortunately though, to me, it seems clunky to use. It's not something that I'd like to write by hand. It is logically very similar to defining Java builds with Ant XML files, if that helps. You can get the job done, but if you want to have any sort of complexity or logic, it's not going to be easy. Any XML/JSON/YAML format is going to have this issue.

I think the jury is still out though as to how much it gets used outside of a few large academic centers. I see it as mainly helpful for large-scale analysis projects where you have to manage cloud resources. Not many people have that problem. I'm personally only planning on using it as an output for my own language (which is basically a Make-like format that submits dynamic jobs to HPC clusters). I have yet to see how WDL can work with dynamic workflows (where you might have more than one path to an output or change the workflow based on a command like argument). It might catch on, but I've seen this show before.

And to answer your question about if these languages are actually used... absolutely! We use WDL extensively, but it is relatively new, only about 1 year old. We've had a lot of success using it so far.

What does WDL bring to the table? Well, WDL is essentially Makefile on steroids, designed for cloud computing. We want to be able to take a shell script and turn it into a function with parameters and return values. Like this: https://github.com/broadinstitute/wdl/blob/gvda_add_scripts_.... WDL calls this a "task".

Then, once we define a bunch of tasks, we need to be able to link them together. Like, have a sub-set of one WDL task's outputs be the inputs to another task. WDL calls this a "workflow". Here is an example of invoking the above task: https://github.com/broadinstitute/wdl/blob/gvda_add_scripts_...

The workflows we're creating are far too large to be run on a single machine, so we wanted a way to specify what each task needs in terms of CPU/memory/disk so we can provision machines that are capable of running each task. Parallelism is also a very important aspect of WDL, and all workflow languages/engines.

As others have said, this is not a new idea. WDL's primary goal is to make this as easy as possible for people to read, write, maintain, and extend workflows. We knew that these "workflows" would be code and they'd evolve, they'd have to be code reviewed and diff'd, and there'd be pull requests of WDL files, so we needed a way of describing them that would work well in this environment.

Workflow systems are generally used to orchestrate parallel batch jobs on a cluster, which may have hundreds or thousands of individual tasks. So the emphasis is on defining what units of work are independent that can run on separate nodes without needing to communicate with other tasks, and how outputs of one task feed into the inputs of another.

A "workflow engine" could encompass the definition of a graph of tasks that need to be run, the monitoring of those tasks, retry logic for failures, alerting for monitoring and some form of UI for viewing progress.

I understand your frustration, but calling us assholes and lazy is just a little harsh. At this point, I'd have hoped it was understood by everyone that this wasn't the sysadmin's fault. How insistent or pushy or hoity toity are you were when asking? I've been woken enough times while on call after a 2.7 upgrade to pretty much give up on it, so even the request gives me a blinding headache these days. So yeah, I'll probably be a little grumpy at that request and I will take my time with it on top of the other things going on. If I was called lazy or an asshole for bringing these issues up, be ready to wait.

Hell, to flip it back on y'all, would it hurt to have proper shebangs? For every developer who asks for a 2.7 upgrade, I get three more asking to downgrade until they fix their badly self-deployed code, despite plenty of preparation. It sucks pretty hardcore.

That said, I have a flood of baseline rhel/centos machines to install soon, so I'll try to include it in my builds.

(most of these issues I've had are in larger environments where best practices are best-effort, through legacy swamps.)

"The problem with make for scientific workflows":

http://bionics.it/posts/the-problem-with-make-for-scientific...

"Workflow tool makers: Allow defining data flow, not just task dependencies"

http://bionics.it/posts/workflows-dataflow-not-task-deps

The last of which is a limitation of even the most of the "very much-engineered" ones, as the post goes on to explain.