Reactive, reproducible, collaborative: computational notebooks evolve

Lots of people seem to state that git already solves lots of the versioning issues for ML experiments. However I am yet to find a good guide or example.

Just tagging commits and experiment runs does not seem enough of a framework to manage large experiments.

Many existing version control tools help with this, but there are some challenges that crop up when doing this with jupyter notebooks:

1. Versioning notebooks in a semantically-useful way is difficult: think of it more like versioning binary data than source code. I know we struggle with this a fair amount.

2. There are many debates, some here on HN, about "best practices" for collaborative versioning (do you squash? rebase? etc). These questions exist for notebook users too, and there's a lot of interest in developing tools that either make these decisions for you (are opinionated), integrate them more into the GUI nature of notebooks, or both.

Hi, one of the creators of Vizier here.

The difficulty with git in this use case is that it relies on the user to manually commit. If you try the analysis, and decide to throw the code away before you commit, too bad! There's no longer a record of it.

Vizier records everything. Every single change to the notebook, just like track changes in your word processor/spreadsheet of choice. Speaking from experience, it's really nice not to have to think about manually check-pointing... the history is just there.

Never tried but been using Colab pro and for most part it works pretty well even though I use it for R kernel. The most difficult part in such notebook environment is the installation of some non-standard R packages such as "Vegan" and "Phyloseq" but otherwise no major issues.