Niex: Jupyter Notebooks but Using Elixir
github.com
github.com
Jupyter has an API for writing your own storage back-end, and I whipped up a prototype to store as regular python files, except adding a space after the first character in any line matching /#[ ]*#/ and putting all of the Jupyter metadata on lines beginning ##. Stripping out the metadata and getting back the original formatting means just removing all lines beginning ##, and removing the second character in every line matching /#[ ]+#/. Unfortunately, under my setup, I had lots of trouble with Jupyter timing out its calls to my storage back-end, likely due to running on a heavily loaded virtualized host.
Working directly with the source files is just one of the niceties it has.
1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see.
2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order.
3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified.
Wow. This is the way.
The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything.
I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.
It works a lot better to only keep the markdown in source control. The downside is that anyone who starts from a clean repository scratch will need instructions on how to generate the .ipynb files.
Of course it doesn't know that you only care about the code, but the diffs are a lot cleaner to read. There's also a version of the diff that will basically create a nicely formatted notbook for viewing differnces in a browser.
(Disclaimer: this tool was written and maintained by colleagues of mine.)
In practical terms, it means its easy to hack together your own internal custom notebooks for whatever purposes you'd like (if you know Elixir & Phoenix).
- Collaborative notebooks.
- Long-running notebook scheduling:no more oops, I closed the browser tab.
- Automatic experiment tracking: detect and save metrics, params, and model so you don't have to remember to do it or pollute your code.
- Deploy a model in one click to a REST endpoint, and monitor your model's performance. Or package it into a Docker image and push it to a registry.
Here's an invite link: https://iko.ai/invite/lEMzE_hKwJ2SUbfLdnK7SbZb1c3zUCOAQexakL...
A notebook server being a heavily asynchronous application, it makes a lot of sense to build one using Elixir. It would be a dream come true if Niex could be compatible with the Jupyter kernel protocol to integrate with kernels that already exist for other languages and also compatible with Jupyter frontend extensions. Basically it could replace the Tornado app while leveraging the rest of the existing Jupyter ecosystem. Easier said than done obviously ;-)