Reflow, a language for distributed, incremental data processing in the cloud

If you're into workflow runners, Reflow and Cromwell (https://github.com/broadinstitute/cromwell) are the only two really to consider. Having tried them all, these two are by far the best and most supported (and there are 100s!)

Cromwell is great because it is google cloud native and supported within the Terra ecosystem (https://app.terra.bio/) meaning you do not need to host it yourself - you can just connect your google account and go.

Reflow, I've heard, is a little more "professional" given that the Grail team is heavily ex-Google. But both can scale to massively parallel (1000+ parallel analyses).

From the README:

Reflow comprises:

- a functional, lazy, type-safe domain specific language for writing workflow programs;

- a runtime for evaluating Reflow programs incrementally, coordinating cluster execution, and transparent memoization;

- a cluster scheduler to dynamically provision and tear down resources from a cloud provider (AWS currently supported).

and

Reflow was designed to support sophisticated, large-scale bioinformatics workflows, but should be widely applicable to scientific and engineering computing workloads. It was built using Go.

Reflow joins a long list of systems designed to tackle bioinformatics workloads, but differ from these in important ways:

- it is a vertically integrated system with a minimal set of external dependencies; this allows Reflow to be "plug-and-play": bring your cloud credentials, and you're off to the races;

- it defines a strict data model which is used for transparent memoization and other optimizations;

- it takes workflow software seriously: the Reflow DSL provides type checking, modularity, and other constructors that are commonplace in general purpose programming languages; because of its high level data model and use of caching, Reflow computes incrementally: it is always able to compute the smallest set of operations given what has been computed previously.

Interesting to see Grail share this, I'm excited to try it out.

I'm perpetually unsatisfied with bioinformatics workflow software. Snakemake and GNU make remain my favorites so far in terms of developing novel analysis. However, making GNU make into a reusable pipeline always feels like an awful and ugly hack. And GNU make requires a shared file system among nodes, which is problematic on AWS...

This seems to have potential for both recording the steps for reproducible science, but also turning those set of steps into a reusable pipeline easily.

Is reflow go only?

is the .rf file format a DSL or an existing language?

One thing I don't get is its claim of being referentially transparent. How can this possibly be the case in the presence of exec commands that call shell scripts (which could do anything, potentially nondeterministically)? Or does the type system take this into account somehow?

This looks cool, but for companies tied to k8s, probably a no go? I'd be all over this if it could just run on our k8s cluster..