Nextflow: Data-Driven Computational Pipelines | Hacker News Reader

Nextflow: Data-Driven Computational Pipelines

nextflow.io

90 pointsbrianzelip40 comments

I develop bioinformatics pipelines for a living and am very opinionated on the topic.

Having enough experience with snakemake as well as nextflow in production for many years now, I would always opt out for snakemake for anything but extremely large DAgs (which is quite rare for for bioinformatics pipelines). The fact that nextflow still doesn't allow deleting temporary files during execution or re-rerunning the workflow from a set of intermediary files is an insane deal breaker (not mentioning other strange things like an arbitrary limit of 1000 parallel jobs etc.). AWS runners that once were nextflow's selling point is not an advantage anymore give Amazon Genomics Cli.

Subjectively, writing pipelines that incorporate conditional logic is much nicer in python+snakemake than groovy+nextflow, but maybe there is someone out there who prefers groovy.

Is a proper dry run possible in nextflow already btw?

I do too.. and have similar opinions. I wrote my own tool years back for pipelines because it was always frustrating (started roughly around the same time as Nextflow).

Allowing for files to be marked as transient (temp) and re-running from arbitrary time points are definitely one of the things I support... as is conditional logic within the pipeline for job definition and resource usage. For me though, one of the biggest things is that I like having composable pipelines, so each part of the larger workflow can be developed independently. They can interact with each other (DAG) and use existing dependencies, but they don't have to exist in the same document/script. I work on large WGS datasets, so 1000's of jobs per patient isn't uncommon.

Happy to talk more if you're interested.

https://github.com/compgen-io/cgpipe

And yes, you can dry run the entire thing. It will write out a bash script if you want to see exactly what is going to run without submitting jobs. It's a full language for pipelines, but heavily inspired by Makefiles (with conditional logic).

Interesting, thanks for sharing. I'll definitely take a look, although at this point I am so comfortable with Snakemake, it is a bit hard to imagine what would convince me to move to another tool. But I like the idea of composable pipelines: I am building a tool (too early to share) that would allow to lay Snakemake pipelines on top of each other using semi-automatic data annotations similar to how it is done in kedro (https://github.com/kedro-org/kedro).

You can resume. But the caching is very finicky, especially when processing groups of files. I often have to sort tuples or set caching to “deep” to get resume to actually do its job.

The fact that they have a whole blog post called “demystifying nextflow resume” suggests it isnt intuitive for most.

It's been a while since you can rerun/resume Nextflow pipelines, and yes, you can have dry runs in Nextflow. I have no idea what you're referring to with the 'arbitrary limit of 1000 parallel jobs' though. As for deleting temporary files, there are features that allow you to do a few things related to that, and other features being implemented.

> It's been a while since you can rerun/resume Nextflow pipelines

Yes, you can resume, but you need your whole upstream DAG to be present. Snakemake can rerun a job when only the dependencies of that job are present, which allows to neatly manage the disk usage, or archive an intermediate state of a project and rerun things from there.

> and yes, you can have dry runs in Nextflow

You have stubs, which really isn't the same thing.

> I have no idea what you're referring to with the 'arbitrary limit of 1000 parallel jobs' though

I was referring to this issue: https://github.com/nextflow-io/nextflow/issues/1871. Except, the discussion doesn't give the issue a full justice. Nextflow spans each job in a separate thread, and when it tries to span 1000+ condor jobs it die with a cryptic error message. The option of -Dnxf.pool.type=sync and -Dnxf.pool.maxThreads=N prevents the ability to resume and attempts to rerun the pipeline.

> As for deleting temporary files, there are features that allow you to do a few things related to that, and other features being implemented.

There are some hacks for this - but nothing I would feel safe to integrate into a production tool. They are implementing something - you're right - and it's been the case for several years now, so we'll see.

Snakemake has all that out of the box.

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge.

At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.

Other reasons I'd like to start using Nextflow:

- my homebrew pipeline would be easier to setup/share

- there are some efforts in the proteomics community to develop Nextflow pipelines (eg. QuantMS²). I think it would to have a shared language to express pipelines, and it would make benchmarking simpler.

___

¹ https://github.com/coleifer/huey/

² https://docs.quantms.org/en/latest/

From one proteomics person to another, what tools are you using? I can see needing snakemake for something like proteogenomics (our lab published a tool in that area) or DIA cause that pipeline can get a little complex. But for run of the mill stuff, as long as I have CLIs, I don’t really find myself needing anything beyond a basic batch file.

Before you wonder why I don’t know that, I do top-down software development primarily and also maintain and upgrade my lab’s search engine, MetaMorpheus.

The DDA pipeline goes like this: ThermoRawFileParser -> Comet -> [a bunch of OpenMS tools] -> Percolator -> [custom quantification stuff]. The data is mostly derived from chemoproteomics experiments where you have isotopically labeled control and compound treated samples that are enriched with some probe. As a result, we work a lot with ratios and have to differentiate the scenario where your compound completely blocks probe labeling/enrichment or it's just stochastically missing due to the nature of DDA. For TMT it's pretty similar. I'm working on DIA as well though it turns out there's still quite a few challenges there for our particular use case.

To answer your broader question about the general need for some structured pipeline or workflow orchestration.. That comes down to volume of data (we do screens as well as one-off studies) and a desire to reduce human involvement as much as possible. So the goal is to have raw files be immediately picked up, processed, and loaded into on internal application where it can be queried and interesting data can be highlighted. During my PhD, this was also a goal of mine (and I have at least two github repos where I got close) but it was definitely less of a priority since actually doing experiments and downstream analysis was the limiting factor.

PS: if you want to talk off-HN, I should be your latest stargazer

I’d be happy to chat offline. Your current work and my current work I think share a lot of similarity. My GitHub is avcarr2.

The closest I’ve gotten to local debugging is having the Python scripts that are launched by NextFlow steps connect to a remote debugger process (“remote” but running on the same workstation). PyCharm makes this fairly painless to orchestrate. I’ve never been able to debug thr Groovy script in a Nextflow pipeline itself; I think you’d need a debug build of the nextflow executable for that.

As someone who dips into nextflow from time to time, I'd strongly suggest by developing your pipeline based on an existing nf-core pipeline or the nf-core templates. nf-core comes with a bunch of nicer defaults like profiles for SLURM, Singularity, Docker that help you abstract some of the headaches away, plus you could get lucky and can just glue some of their modules together.

I don’t know. I started this route and then quickly switched to only dipping into nf-core when they had actual prior art.

The interplay of nf and groovy (how I wish they hadn’t used groovy!) can be mind bending but if you’re writing your own thkng you have a different optimization model than nf-core that is trying to be one size fits all

You can debug snakemake with pdb. It also has actual dry-runs to test the dag before actually running anything (with nextflow you have to test run with “stubs”)

Another day, another workflow DSL.

  * looks like yaml
  * has curly braces to look programmery
  * whitespace might be meaningful
  * has pipes like a bash script

https://www.commonwl.org/

https://github.com/common-workflow-language/common-workflow-...

mea culpa: The above was based on a first look at something titled "A DSL for parallel and and scalable computational pipelines"a as opposed to "Java workflow manager with Groovy language scripting." The presented screenshot still looks to me like an unholy union of yaml, js, py and sh. If that sounds groovy to you; have fun.

In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25

As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right way to scratch.

However all of them were rejections of prior models as well as the workflow solutions prominent in the business space.

They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.

While true that’s a minor distinction when comparing the clusters of bioinformatics workflow systems vs workflow systems aimed at different domains

While these two are aimed at bioinformatics, they are general purpose enough that you can apply them to any computational workflow. I can say they saved my PhD

Yeah, the thing that I find disappointing is that there is a lot of science value locked into the different systems of describing a workflow, pipeline or DAG. Like you said, they all had different itches to scratch and even some barebones "standards" like csv have flavors/extensions/etc.

In fairness, this is an old problem with many other contenders. This issue is as old as batch schedulers. FWIW, I was at an ISMB conference in 2005 that had at least 2-3 workflow managers presented.

If you refer to nextflow,the syntax is basically groovy

Indeed, it's a fully functional scripting language that is an extension of groovy. So "looks programmery" is more than a little reductive.

Edit: functional as in, it's not half-hearted, not functional as in functional programming.

Interesting. From the docs:

The Nextflow scripting language is an extension of the Groovy programming language. Groovy is a powerful programming language for the Java virtual machine. The Nextflow syntax has been specialized to ease the writing of computational pipelines in a declarative manner.

https://www.nextflow.io/docs/latest/script.html?highlight=gr...

Its really just a layer on top that uses the groovy DSL features

http://docs.groovy-lang.org/docs/latest/html/documentation/c...

I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once.

I've used nextflow in the past, and I've found it to be much harder to use. Learning another DSL is annoying, documentation was sparse, I constantly ran into bugs, and it was hard to debug in general. I don't know how much it's changed over the past 3 years though.

¹https://github.com/insitro/redun

This is the first time I've seen this and it looks very interesting -- thank you for sharing!

I described my workflow in a different comment on this post, and this seems like something I could port to with minimal changes in code since every step is already a Python function and even decorated by @task.

In theory I want to like Nextflow, but my main criticism is that it's really, really hard to debug programs that pass around lists of Promises (nextflow's dag uses promises as handles on future computations, and functions receive promises and can't easily materialize them and print them.

The caching is often more trouble than it's worth. Also, the little bash scripts that integrate with AWS break if your AWS environment isn't vanilla (our enterprise AWS has a lot of restrictions).

Using Tower for Nextflow can help streamline it on AWS, it’s pretty powerful (but costs money for anything beyond trivial use cases): tower.nf

It's kind of a shame this is based on Groovy, rather than Python which is much more familiar to people in the HCLS space. I've always been stuck on the fence between wanting to use NF (since it's the most popular) and Snakemake (which feels like less of an oddity development-wise).

Perhaps it is not as popular, but I found the groovy syntax ideal for DSL like that

I feel like groovy pushes towards the worst of both worlds between an internal dsl and external dsl. It’s an internal dsl so you get the language but oh man groovy sucks

So my question to the non bioinformatics - is this already a solved problem?

You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on).

Something like dask perhaps,but can specify a docker image for the task?

What is the goto in DevOps for similar tasks? GitHub actions comes pretty close...

To bioinformatics what is the unique selling point of next flow over say wdl/Cromwell?

I do computational physics and I use Snakemake. On HPCs, we only have user-level access. We are not allowed to perform long-running processes on login nodes, could be killed at any time due to a violation of rules. That said, anything that depends on Docker is a no-go; anything that uses a server-client structure we will try to avoid (although it might be possible for us to host a daemon elsewhere that we as students pay out of our own pocket). We also deal with a lot of tools that are not well-written in Python or modern languages, you wouldn't want to build any CFFI onto it.

So Snakemake, and similarly, Nextflow, suits our needs well. It is a user-space CLI tool that does not require any privileges, it optimizes for running bash command / any CLI-based tools. A bonus for Snakemake is that it uses Python and our other scripts use Python too.

So I guess DevOps tooling, which heavily bias towards docker or whatever container-based execution, is really a different space.

The big difference when comparing bioinformatics systems with non are what the typical payload of a DAG node is and what optimizations that indicates. Most other domains don’t have DAG nodes that assume the payload is a crappy command line call and expecting inputs/outputs to magically be in specific places on a POSIX file system.

You can do this on other systems but it’s nice to have the headache abstracted away for you.

The other major difference is assumption of lifecycle. In most biz domains you don’t have researchers iterating on these things the way you do in bioinf. The newer ML/DS systems do solve this problem than say Aorflow

I for one have started to appreciate the fact that the shell/commandline interface means:

- We have an interface that very strongly imposes composability, that is rarely seen in other parts of IT, and making people actually "follow the rules" :D

- Data is (mostly) treated as immutable, except perhaps inside tools

- Data is cached

- The cli boundaries means that at least one can inspect inputs/outputs as a way to debug.

- Etc...

Personally, the biggest frustration is all the inconsistencies in how people design the commandline interfaces. Primarily that output filenames are so often created based on non-obvious and sometimes arbitrary rules, rather than being specified by the user. If all filenames were specified (or at least possible to specify) via the CLI, pipeline managers would have such an enormously easier time.

What happens now is that you basically need a mechanism like Nextflow has, where all commands are executed in a temp directory, and the pipeline tool just globs up all the generated files afterwards. This works, but opens a lot of possibilities for mistakes in how files are tracked (might be routed to the wrong downstream output, if you do something funny with the naming, such that two output path patterns overlap).

nextflow can't even get this right- base nextflow uses some combination of `--paramName` and `--param-name` and treats them as interchangeable, while nf-core encourages `--param_name` (but nextflow sees that as different). All trivial differences but just layers on the CLI frustration train.

How many workflow management systems do we need? Over 300[1]. If that's not reinventing the wheel.

[1] https://github.com/common-workflow-language/common-workflow-...

Pretty much everyone underestimates the complexity of workflow scheduling.

This means like 99% of tools have had various random limitations and quirks that make them not applicable to a lot of use cases.

This is something that people using these tools mostly never realize (and so complain about the "reinvention").

The folks I know who tried to implement a pipeline tool are often a bit more aware of the challenges and how hard it is to make something that is really general.

I say this as someone who evaluated a dozen tools, and finally extended an existing tool to fix some limitations (Luigi, with our SciLuigi extension), and finally developing our own tool (SciPipe).

It has gotten better, and a tool like Nextflow is pretty generic these days, although they also might have limitations. For example, before DSL2 we needed re-usable modules, which is why we developed SciPipe, which otherwise has a very similar scheduling mechanism to Nextflow (Dataflow/Flow-based).

Still today, I'm having mixed feelings about using extremely complex tools that are dependent on a single organisation to keep updating. Not being able to easily debug execution and a few other things, which is why we wanted a simple library that we could understand ourselves and run through a debugger. (And it didn't hurt that we could get complete audit logs per output file, which can be very useful both for provenance and debugging, and is not found in almost any other tool.)

Just to give some examples of why someone might still entertain thoughts about developing separate tools.

All in all, the widely used ones like Nextflow (and Snakemake) are great tools. They just aren't optimal for every usecase and situation.

Reply on news.ycombinator.com