The Jupyter+Git problem is now solved (2022)

> Alternatively, if you don't care about cell outputs then Jupytext[5] to sync .ipynb JSON to markdown

Notice that using markdown is a possibility for jupytext, but not the only one. More interestingly, you can also store your notebooks as plain python files, whose comments are interpreted as the markdown cells of the notebook.

This is very useful, and not only for version control: if your notebooks are python files they can be executed easily in CI or by third parties just by launching the interpreter. No need even of the jupyterlab dependency.

With some care, you can craft a single python file "foo.py" that can be used at the same time as

1. an executable command-line program (that happens to be written in python)

2. an importable python module

3. a jupyter notebook (to open it you need the jupytext extension of jupyter)

4. the documentation with auto-generated figures, convertible to html or to pdf using "jupyter nbconvert --execute"

5. a regular .ipynb if for some reason you want to distribute the outputs in a re-executable format

For small simple projects, to showcase, describe and illustrate an independent algorithm, we have found this structure invaluable.

This is a post from my Linkedin page on my hopes for Jupyter notebooks and git. Anyone know of progress along this line?

#Jupyter notebook and git

As much as Jupyter Notebooks have been a great tool for data science, the transition to deployment, and the general software engineering friendliness of Jupyter Notebooks could use some work. From time to time, I have explored how others have dealt with turning notebooks into an organized codebase and outputs. To date, I have not found a comfortable approach for me. The ideal approach for me would be to use something like 'node metadata' in the way of [Leo Editor](https://leo-editor.github.io/leo-editor/) to function as 'decorators' for a notebook cell for integration with git.

By this I mean using something like special markers in Python comments (since much of data science is done with Python) to map the content of a cell (or output) to a git repository. Better yet, define a special cell type for git metadata preceding a code cell. Then implement some basic git operations on the contents of a cell. Let's suppose we use @@git as a marker for metadata in comments for git. --- beginning of cell --- # @@git %upstream%=https://github.com/pyro-ppl/pyro # @@git %local%=~/repo/pyrodev # @@git %branch%=burnburnburn # @@git %file%=examples/cvae/util.py

# Here begins the contents of the util.py file ... --- end of cell ---

An extension would implement items in the menubar for various git operations: stage - stage the content as util.py file checkout - checkout from upstream, replace local copy, and refresh content of cell commit - commit stage file specified by %file% status - ...

Imagined workflow is that once a working idea scattered throughout a notebook has been sketched out, the user would mark the notebook cells that should be mapped to files in a git repository. Also this could be used in a mixed dev/data science environment where library code under development can be pulled right into a notebook.

Yes, there will be problems with committing code with comments that are specific to one user which is why a special cell type makes sense. Yes, there will be problems that I can't even imagine right now but ...

Please message me if you know of a cell-based git extension for Jupyter Notebooks.

Totally agree with this. I like the Markdown subset approach LiveBook (https://livebook.dev/) has taken to play nicely with version control in comparison.

I suspect that .ipynb will eventually be displaced by good old HTML documents loading PyScript (or more specifically, a notebook framework built on PyScript).

The premise behind nbdev isn’t that DS software should be a different set of tools - it’s that we should all consider using a different set of tools, based on literate and exploratory programming principles.

I build all my projects, including a SQL lib, an EC2 interface, http and fastcgi frameworks, various web apps, and a mail merge system in notebooks, in nbdev and it’s made me much more productive.

Amongst folks that have used nbdev for 1+ years that I’ve spoken to, all report a 3+ multiple of improvement in productivity (based on non rigorous self assessment). This could however be biased because these people also report enjoying coding much more, so it’s possible some of that effect is the time just doesn’t seem as long.

Personally, after coding previously for over 20 years in various IDEs and editors (and for instance being prolific enough in vim I often gave talks about it) I wouldn’t ever want to go back to that old pre-notebooks time.

But it’s a huge investment and requires a lot of relearning - to get the most out of it you have to rethink just about everything. So it might be better for those earlier in their careers that haven’t got as many habits to change.

Why shouldn't they use nbdev if it works for them? Plain Git obviously does not work and not only for data scientists. Git, with its inability to deal with non-textual artifacts, is completely unsuited to a modern software development workflow. Modern software development contains interactive graphs, audio and video assets, editable diagrams, vast amounts of binary data for reproduction of experiments, etc. etc. So much custom tooling has been built to somehow shoehorn this stuff into git and it never works well. I'm sick of diagramming software and IaC tools with horrible user experience just to keep a plaintext representation that no human dares to read anyway.

ive written two projects with nbdev - its great, but after a while we moved off of it - too limiting

Strongly agree with this. IIRC, in RMarkdown state is treated as a separate cache stored outside the notebook and loaded as needed. You could use something like dvc or gitlfs to manage those cache files, and since the Markdown file is plain text, use regular git to inspect changes to the notebook implementation.

I feel like Jupyter notebooks are the PDFs of data science. They are super useful for displaying results, but bake that data in a super inconvenient way for doing anything but rendering the data to look nice.

You want to share the outputs so that you can share the results while showing your work. That jupyter also inlines everything, such that even charts are stored in the document, makes that even more necessary.

Though, I can see your point, I think. Why not include a build step that moves from your document to the generated output? My gut there is a large part of why the system got popular is that they worked hard on removing the friction that that would add.

As a comparison and to your point, I've seen people try to build "literate test suites" that were in a notebook, very happy with how the output looked. Only to find later that if they had used some of the more common test frameworks, those already create very nice reports. And moving the report format/creation out of the specification allowed a ton of flexibility.

> I'm curious why one would specifically want to save cell outputs as is in the Jupyter notebook

My blind guess is that it improves the readability of the notebook / promotes the literate programming mindset. But just a guess

No the answer was simply that git doesn’t come with a merge driver for json, only one for line oriented text. So nbdev provides one for ipynb json documents.

Yes, I think you are right. Seems like quite a good plan to me. Nothing else obvious jumps to mind, as anyone using git is going to end up with git conflict markers in their files at some point.

yeah it's definitely some classic engineering around the problem, but hard to fix Github I guess. At a previous company, we switched to Gitlab for the specific reason that they handled notebook diffs slightly better. It's a struggle out here

Thanks, but this is a bit pedantic. For most people Jupyter is the notebook. You could just as well call the other one the "Jupyter protocol".

What is the difference between the two? Googling it shows some comparisons between jupyter lab vs notebook but I can't find anything on plain jupyter.

There isn’t a problem any more with notebooks and git because nbdev now provides a merge driver. So it’s fine to use notebooks with git now.