Hacker News Reader

Comment by jerrygenser | Hacker News Reader

I see it can be used to define and run workflows. But for reproducibility on top of the execution of operations on data, I'm wondering if there's any way to version the underlying data?

Or would you typically use this -in addition- to a tool like dvc? I've used dvc a bit and while it's quite good for data versioning, I find the workflow aspect is clunky.

View on HN

The way a workflow like Snakemake can help here is generally by letting the filenames pretty much describe how each particular output was created, meaning data outputs can act as immutable in a sense.

What I mean is that rather than create a new version of a file, if you run the same analysis with different sets of parameters, it should generate a new file with a different name rather than a new version of the old one. This also helps comparing differences between output from different parameters etc.

That said, there are workflow platforms which support data versioning, such as Pachyderm (https://pachyderm.com), but it is a bit more heavyweight as it runs on top of Kubernetes.

The reliance on filenames to define (parametric) dependencies was among the reasons I later adopted nextflow. The model fit the type of computation dependencies better for my case. In the mean time snakemake grew and many DAG that were hard to describe back then are now expressed directly with snakemake primitives

I am very fond of https://zenodo.org, especially for small datasets for scientific publications.

There is indeed an old issue asking whether they could return a provenance graph using the prov ontology in JSON format https://github.com/snakemake/snakemake/issues/2077 I think it's a good task to work on if you like to contribute

I use snakemake along with dvc. It detects changes in files, and reruns steps to produce downstream files as needed.

Pachyderm has data versioning / data version control built-in. I guess other tools do too.

Reply on news.ycombinator.com