Hacker News Reader

Comment by frgtpsswrdlame | Hacker News Reader

Wow this looks perfect for what I need right now - just a bit of lightweight tracking.

DVC also fills the "lightweight tracking" niche, although it relies on automatically creating Git branches as its technique for tracking experiments. I personally find that distasteful, so I don't use it specifically for experiment tracking, but the feature is there.

The company behind DVC is also building a handful of other related tools, e.g. https://iterative.ai/blog/iterative-studio-model-registry

Hm, in what way do you find that DVC requires creating new branches for experiment tracking?

I find the following workflow works well, for example:

1. Define steps depending on a `config.yml`.

2. Run an initial experiment (with an initial config) and commit the results.

3. Update config (preserving the alternate config and using symlinks from `config.yml` to various new configs if necessary), re-run, and commit.

4. Results are then all preserved in your git history.

Right, but experiments aren't always linear. Do you really want to make a new commit for every iteration of a hyperparameter search? What if you are using a black-box optimizer that supports parallel/concurrent updates?

I don't want to use Git to track all that. I want to use Git to store the final results of running such an experiment in the same commit as the code that implemented it. I just don't like the DVC experiment workflow, but I am more than happy to use DVC for storing the fitted model(s) at the end of the run.

Yeah, that's a fair point, and I agree. I don't think it's ideal.

If you use `dvc exp run` you don't need to commit anything as I mentioned above. You can run multiple experiments in parallel, etc. Commit happens only / when you want to select the best result and share it with the team. But even that is optional.

It doesn't require creating a branch when you iterate, it requires creating a branch or commit if you want to share it with the team - see it on GitHub or in Studio. But even those lightweight iterations (https://dvc.org/doc/command-reference/exp/run) could shared as well via Git server - they won't be visible for now via UI in GH/Studio at the moment.

Happy to provide more details on how it's done. It's actually quite interesting technical thing - custom Git namespace https://iterative.ai/blog/experiment-refs

If you need help, you can open an issue on GitHub (https://github.com/ploomber/ploomber-engine) or join our Slack! (https://ploomber.io/community/)

Reply on news.ycombinator.com