Hacker News Reader

Comment by KMag | Hacker News Reader

I haven't looked at the storage format here, but for anyone looking to make a Jupyter-allike, please make the storage format friendly for code unmodified code review systems. Jupyter stores python inside JSON, and you can read the diffs, but it's a bit painful.

Jupyter has an API for writing your own storage back-end, and I whipped up a prototype to store as regular python files, except adding a space after the first character in any line matching /#[ ]*#/ and putting all of the Jupyter metadata on lines beginning ##. Stripping out the metadata and getting back the original formatting means just removing all lines beginning ##, and removing the second character in every line matching /#[ ]+#/. Unfortunately, under my setup, I had lots of trouble with Jupyter timing out its calls to my storage back-end, likely due to running on a heavily loaded virtualized host.

View on HN

I think every new notebook project should take ideas from Julia's Pluto: https://www.youtube.com/watch?v=IAF8DjrQSSk

Working directly with the source files is just one of the niceties it has.

Thanks again for the pointer. Pluto really seems to nail the major pain points from notebooks: for a quick summary of Julia's Pluto from https://lwn.net/Articles/835930/ :

1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see.

2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order.

3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified.

Wow. This is the way.

My only complaint about the pluto system is it's use of strange extended ascii characters.

That's a Julia thing, not a Pluto thing. Not a fan personally but I think that's a minority opinion in the Julia community.

I'm fine with Julia's use of unicode for math symbols in general.

The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything.

I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.

Very interesting. Thanks for the pointer. LWN has a nice textual introduction: https://lwn.net/Articles/835930/

Jupytext makes this all a lot less painful by keeping a Jupyter notebook up to date with a markdown counterpart (without output).

It works a lot better to only keep the markdown in source control. The downside is that anyone who starts from a clean repository scratch will need instructions on how to generate the .ipynb files.

Can you customize the diff command used by your code review software? There is a diff tool for Jupyter called nbdime you can install with pip.

Of course it doesn't know that you only care about the code, but the diffs are a lot cleaner to read. There's also a version of the diff that will basically create a nicely formatted notbook for viewing differnces in a browser.

As far as storing a human readable version for code reviews and diffs, we use nbautoexport for this - https://nbautoexport.drivendata.org/ - which automatically exports plain text source files.

(Disclaimer: this tool was written and maintained by colleagues of mine.)

This tool looks cool. Unfortunately, it looks like this tool in export-only. Ideally, one wants a single source of truth, and if your reviewed artifacts can't be imported for execution, then your code review can't be reviewing your source of truth. Don't get me wrong: it's much better than reviewing JSON diffs, but in highly regulated/conservative industries, we'd prefer to be reviewing the actual code that gets compiled/interpreted.

How do KDE Cantor's project files compare to Jupyter? I want to try out Cantor next time I want to work in a notebook.

This does the same thing. An example is on the main page of the repo: https://github.com/jonklein/niex#notebook-format

R notebooks get this right. They are written in Markdown. And they support Python and friends too.

Reply on news.ycombinator.com