I've used Snakemake my whole life, can someone experienced with both systems share whether jumping to nextflow is worth it?
For example, snakemake makes it very difficult if not impossible to create pipelines that deviate from a DAG architecture. In cases where you need loops, conditionals and so on, Nextflow is a better option.
One thing that I didn't like about nextflow is that all processes can either run under apptainer or docker, you can mix and match docker/apptainer like you do in snakemake rules.
you meant "CAN'T", right?
1. A process that "generates" protein sequences
2. A collection of processes that perform computationally intensive downstream computations
3. A filter that decides, based on some calculation an a threshold whether the output from process (1) should move to process (2).
Furthermore, assume you'd like process (1) to continue generating new candidates continously and independently until N number of candidates pass the filter for downstream processing.
That's not something that you can do easily with snakemake since it generates the DAG before computation starts. Sure, you can create some hack or use checkpoints that forces snakemake to reevaluate the DAG and so on, and maybe --keep-going=true so that it won't end the other processes from failing, but with nextflow you just set a few channels as queues and connect them to processes, which is much easier.
That said, I do find snakemake easier to prototype with. And it also has plenty of production features (containers, cloud, etc). For many use cases, they’re functionally equivalent
Technically snakemake can do it all. But in practice NF seems to scale up a bit better.
That said, if you don’t need the UI for scientists, I’d stick to snakemake.