Hacker News Reader

Comment by mxuribe | Hacker News Reader

Beyond the cool python angle, I find the following statement quite interesting: "...James Somers argued that Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." I've only been exposed to jupyter notebook references here and there...but i guess i should become a little more familiar with them.

View on HN

While I think this is a good thing, an additional requirement is needed. The published notebook should be sequentially executed by an automated tool. Users can execute commands out of order and have artifacts from deleted commands. Without verification, you can be publishing notebooks with bugs in them that you don't see until you re-execute them.

Jupyter notebooks are ultimately just a text file, which can be version controlled, verified (via the VCS or independent hashing), and if necessary - digitally signed in a cryptographically secure manner.

Given that Git is already secure, I'd say all the researcher needs to know is basic usage of Git. Version controlling the notebook satisfies all the requirements you mentioned: prevents accidental distortion, is verifiable, restorable, etc.

It's not just tracability and security that's the issue, though. It's an issue of reproducibility. When I give you a notebook, how do you 'run' it to verify my calculations? You can't just press "run", because I might have executed the cells out of order, or deleted cells which later cells depended on. Version control doesn't solve this problem. Good practice might solve it, but it turns out that researchers are stunningly bad at following a strict top-to-bottom execution of notebooks. And who can blame them? The non-linearity of notebooks is one of their features. But when it comes to reproducibility and just the basic mechanics of sharing code, we definitely need a better solution.

You forget that a typical Python Jupyter notebook will import a dozen Python packages.

Versioning that it's still a major problem in the ecosystem.

> Versioning that it's still a major problem in the ecosystem.

There's actually a perfectly good way to solve this problem: a virtual environment with a requirements.txt file.

For research in particular, I'd suggest a Pipfile.lock. That will ensure that the packages installed match exactly, not just by version number.

Tools for combining the interactive work in notebooks with a more predictable linear flow are starting to mature. See for instance jupytext [1]: a bi-directional converter between jupyter notebooks and python/R/julia/... I believe jupytext was featured here on HN a few weeks back.

[1] https://github.com/mwouts/jupytext

I was recently asked by a company to do a take-home assignment, some data analysis stuff. I opted for a jupyter notebook as it sequentially allows you to both explore the data in an interactive manner while still programming in Python. It was great. I love notebooks.

"Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." - hardly a replacement as long as Jupyter notebooks are not self-sufficient files you can download and view offline everywhere. Even if you have up-to-date versions of Python and all the libraries installed you still have to run a server and open it in a browser which is a way more clumsy than just opening a PDF in Okular/Evince/AcrobatReader. I also doubt Jupyter notebooks are easy to view on mobile devices.

You can view them just as easily as you would a pdf. Running the code inside them is what takes extra effort. Which isn’t necessary if you just want to understand the processes being applied.

Can they include pre-rendered code execution results to display when code execution is not available?

Yes, they do by default if I’m not mistaken. The output “cells” are saved alongside the code.

Reply on news.ycombinator.com