Making Git and Jupyter Notebooks play nice | Hacker News Reader

Making Git and Jupyter Notebooks play nice

timstaley.co.uk

168 pointsmana9932 comments

Hi, HN!

Author here. A friend mentioned this was on the front page so I wanted to stop by and make explicit that this advice is OUT OF DATE as far as I'm concerned. It's way too much hassle (I work in a much larger team now than I did then!) and doesn't play well with rebase etc.

These days I either recommend the jupytext approach (not tried it but seems sensible) or personally I just use Sphinx-gallery.

Advantages:

* Plain python files play well with IDE refactoring, Black formatter, etc etc.

* You now have a readymade 'tutorial' page for your docs.

* Files are run with every docs build, so you can configure things to alert you when they're broken.

Disadvantages:

* You end up editing a throwaway notebook file. If you forget to copy-paste your edits back to the source, and rebuild, you have lost your edits. However, this forces me to keep the 'temporary, exploratory' nature at the front of my mind and not allow the notebook code to grow too large before performing some clean-up.

Can you not use https://code.visualstudio.com/docs/python/jupyter-support-py... to export/import?

Things I’ve found in checked-in notebooks:

* database credentials.

* sensitive data

* a whole DataBricks webpage because the person didn’t understand how to export just the notebook.

* collections of notebooks named only what step in the process they are, and literally nothing about what they actually do.

* Whole base64 encoded images and zip files

* packages imported by manually manipulating system environment paths

* multi-processing/multithreading by shelling out and calling new python instances

* good old “don’t run these cells”

Untitled9.ipynb

Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells

Jupytext is great. What the world needs is that it becomes--transparently--the default and we can get rid of the silly un-editable json format.

Thank you for mentioning jupytext. I had read about it but had forgotten the name. https://github.com/mwouts/jupytext

I love jupytext, but i feel like it’s a patch on a problem that should have just been solved. Just change jupyter to work directly in the genereres file format and skip the “pair files” hassle.

Or it could be solved at the git tooling side by introducing Jupyter specific merge and diff tools.

adding git tooling for a specific file type seems like a slippery slope, no? (assuming you are saying that git itself should have this tooling built in - if you mean some sort of addon, fair enough but then everyone who uses jupyter & git needs to install that addon)

I am not suggesting that git itself should ship with a bunch of custom merge utilities for specific file types. git ships with a mechanism that allows custom merge drivers. Setting up custom merge drivers might not be ergonomic right now, but it could have some benefits compared to the transformation approach. For example merge conflicts could result in a valid notebook and it could be manually resolved inside the notebook interface, no need to dive into the text file.

I guess you write hooks then, you don't convert on every commit?

Jupytext works as an extension in Jupyter. Your work is saved as py automatically whenever you work on the notebook. So it's not a conversion at commit time, unlike say nbstripout

Oooohhhh! Nice! I'll install it right away!

I found a good compromise by using VSCode Python extension. You can import Jupyter notebooks as Python scripts and the other way around [0]. If I need to work on a notebook, I prefer working with a Python script and the interactive window [1]. Then I commit both script and ipynb version.

[0] https://code.visualstudio.com/docs/python/jupyter-support-py...

[1] https://code.visualstudio.com/docs/python/jupyter-support-py...

Agree 100%. Only thing I’d add is that the killer feature of the extension is that it allows one to treat ordinary python files as notebooks (without converting!) by 1. Connecting to a persistent kernel instance 2. Allowing the user to use magic comments to delineate code cells within the python file. With these two things, and the ability to export as a real notebook, you get the great experience of a notebook — submitting easily editable cells to a kernel one at a time, as many times as you want — without of all the usual baggage that that would entail. Plus you get to edit in a decent editor (and edit things other than just python files in it) instead of the crap Jupyter forces you to use.

VSCode's solution is awesome. Rmarkdown works the same way, and it makes a lot of sense. Let code live as plaintext, it's native format, and only render results at runtime. Trying to preserve results in the file just makes things needlessly stateful.

Can Jupyter stop using JSON and look for a new structure (not only about the file format but also the data fields representation)? The current format is unordered and contains huge binary blobs which makes it very inefficient and version control is simply a pain.

I believe a new design worth sacrificing backward compability.

A lot of people mentioning what is essentially R markdown files as a better approach.

In .rmd the notebook is just markdown where code segments inside triple backticks can be executed.

That sounds nice and git compatible (and it is), but you lose a lot of the convenience of Jupiter notebook, namely that output isn’t stored together with input.

The nice thing about Jupyter notebooks is you get story, code and results in a single package.

At the beginning of your Rmd file, add a chunk that spins your Rmd file to an R script.

https://www.garrickadenbuie.com/blog/convert-r-markdown-rmd-...

Now you have:

- single file (Rmd, plain text) where you can do any edits, which, when compiled, produces:

- (1) you get story, code and results in a single package: one output file (usually, html), where all inputs and outputs are stored together (or, just the outputs, if you turn off echo'ing, e.g. for producing a report),

- (2) you get the code: another output file, which is a simple R script version of your Rmd file.

Code updates to the Rmd are viewable much easier on the output R script (#2), which is also much more convenient for debugging. For static text updates, I look at the Rmd file. For content updates (e.g. did my data change between the runs?), I look at the html (or Word or PDF etc) file (#1).

With Rmarkdown you have 2 options, both have its own merits

1. save code and output separately, so you can save the output (but not comparing the output versions), or always generate output from scratch when needed (this actually help to ensure reproducibility)

2. or you can use R notebook format, which save the result with document together, in some companion folders.

Output stored with input is precisely the problem for version control. Not only committing generated content but generated content mixed with source code.

the .rmd is just text. But the output is cached separately, RStudio can load the full notebook, code and output. So it seems to me it's only a matter of how to integrate the .rmd with the output data so it can be displayed in interfaces other than RStudio.

Plus you get the option of rendering an HTML which IMO is no different than having a jupyter file.

A year ago, I was frustrated (and surprised) to see one can't do code reviews with Jupyter Notebooks. GitHub diffs for notebooks JSON are super messy.

I set out to build ReviewNB[1], code review tool for Jupyter Notebooks. Turns out a lot of other people had this exact problem. One can see visual diffs & write review comments on notebook cell. Currently only works with GitHub though.

If you want to diff locally (before committing changes), you might like nbdime[2].

[1] https://www.reviewnb.com/

[2] https://github.com/jupyter/nbdime

Note: This article is from February 2017.

Google AI team did a very good presentation and blog post about this and other common problems https://cloudblog.withgoogle.com/products/ai-machine-learnin...

I have some projects based on Jupyter Notebook (e.g. https://github.com/stared/thinking-in-tensors-writing-in-pyt...), and collaboration sucks. Even with git, I need to resort to "don't touch these files, I am working on them right now".

I was thinking about using RMarkdown files (vide https://towardsdatascience.com/version-control-with-jupyter-...) as for them diffs make sense.

Does any of you use this approach, or have insights on how to make it good for collaboration AND visible on GitHub?

Interesting. Is it possible to integrate this as a git difftool instead? Sometimes you want to include the cell output in the repository.

I just clear all outputs before committing code. Works well enough for me, but it's cool to see a more advanced solution!

The problem with that is when there are very long running cells and you want to see the output and keep developing, and committing code as you go along.

Neat project ! Dearly needed. But may i ask, why not use pandoc to convert notebooks to orgmode and git that ?

Anybody aware of a corresponding solution for Mercurial yet?

Reply on news.ycombinator.com