Data Version Control (DVC) for Machine Learning Projects

medium.com

8 points·disintegrator··5 comments

The other day, I wanted to build a version control for data. I googled and naturally I found DVC and some other existing options.

What I don't understand about them is they have a built-in pipeline. it is strange to me, because I would use airflow to handle CI/CD and keeps the version control simple. Mixing things together seems to be against "Do One Thing and Do It Well".

Are those pipelines function like git's pre/post commit hooks?

Hey billconan, I'm the author of that post. DVC pipelines are a definition of stages (e.g. extract data, clean, fit, evaluate and so on...), their inputs, outputs and dependencies. So, not similar to Git hooks, I'm afraid.

DVC provides a command `dvc repro` to actually execute the pipeline definition and doesn't dictate where it's run; so you can decide to run it in your CI environment or pipeline orchestration tool or on GPU-powered hardware.

Thank you very much!

Hey, I'm one of the DVC maintainers. DVC pipelines are very different from Airflow. For me the best mental model is a make file for ML projects. It's a way to describe the project, how to build it (for example to package and an experiment somewhere outside, or embed as a single step in the Airflow DAG). It doesn't replace or compete with Airflow which is more about production to me.

Btw, what do you mean by "handle CI/CD"?

so when I first saw pipeline, the first use case came in mind was data validation and cleaning, like ETL.

In my opinion, what's versioned in DVC should have been cleaned up already. So I would offload ETL to airflow and only version the result of it.

Thank you for your explanation. that makes sense to me. I think it's like the project.json of a nodejs project, where you can define a set of commands, such as npm build, npm run, npm serve ...

Reply on news.ycombinator.com