Hacker News Reader

Comment by whacked_new | Hacker News Reader

I used to use Jupytext a lot for this problem, and think it does a decent job. The main problem with Jupytext is its reliance on BYOD (D for discipline), which is a poor (but possibly best available) solution for human systems.

IMHO the Jupyter+Git problem stems from the ipynb format. Jupytext does it "right" in the sense that you can work in .ipynb, and diff in .md. But as long as the base format is diff-unfriendly, all tools are methods of indirection of format complexity to tool complexity.

That's not to take away from the tool -- it looks great. It also takes the D out of BYOD, which is a win. But I think "solving" it means that anybody who receives an ipynb is able to just look at it out of the box, like plain text, so we're still a ways off.

View on HN

I feel like the BYODiscipline problem with jupytext could be solved with relatively rudimentary text-editor plugins.

I've started rolling my own little plugin utilities and, so far, I have a (very) rudimentary notebook-like interface in plain text.

Combine a proper attempt at such a thing with a good interface to a background ipython kernel system (for which ipython could do with some minor enhancements AFAICT), and you'd basically have the best of all worlds (all plain text editor features including version control and personalisation and customisation, and the iterative advantages of notebook code-cell-based runtimes).

Hopefully, with such a combination functioning well, there'd be an emergent feature that allows one to more easily get interactive with a code-base for the purposes of understanding, debugging or developing it.

Personally, my biggest gripe with Jupyter at the moment is that a few years ago they decided to try to create a quasi-IDE (where they'll probably be beaten by VSCode) rather than improve the general utility of the kernel (or kernel protocol/interface?) and/or the essential notebook UI.

It's a personal gripe, and there's clearly value in the web-first interface they've made with JupyterLab (despite the not insubstantial growing pains that project has faced), but, watching ObservableHQ and Pluto (for Julia) focus just on the core notebook interface, while VSCode have focused on the IDE side and easily incorporated or recreated the now rather old/simple Jupyter Notebook interface, both with success, seem like some vindication on my gripe.

By "essential notebook UI", do you mean the old notebook or the lab interface?

The old notebook was painful to extend in JS compared to writing lab extensions in TS. In Jupyter Lab 3 they have taken questionable steps, but so far I have been able to work around issues.

I was referring to the notebook-like "part" of the interface, where JupyterLab is a notebook interface with IDE-like components wrapped around it (file explorer, terminal etc).

The ObservableHQ interface, for instance, I'd classify as just a notebook interface. IE, individually manipulatable code-cells with a shared runtime.

And yea, JupyterLab is better than the classic IMO. But, until recently I'd say, the notebook part of the interface hasn't gotten much love at all, while there've been steps, due to popular demand it seems, to provide alternative UIs that strip away much of what they've added on top of the notebook (ie, simple mode, and now Jupyter Lite).

I haven't really got experience writing extensions in the old Jupyter notebook, and hardly any with JupyterLab, but my experience with JupyterLab was frustrating because it felt like they really killed the ability to implement small and hacky plugins like you could with with the old. This always struck me as a shame. A necessary one perhaps given what I presume is the increased power of their new framework. But it always felt like there was a mismatch between the complexity of the plugin framework (which is a full web-dev experience) and the base features of the "product", where customising my test-editor is now much easier AFACT.

What issues and questionable steps were you thinking of?

> What issues and questionable steps were you thinking of?

2 things come to mind right now:

Starting with JupyterLab 3 (maybe 3.1), JupyterLab removes query arguments from the URL. Query arguments were the only way I know, to give arguments from the outside of JupyterLab to JupyterLab. Any extension, that relies on arguments given from the outside would break, just because JupyterLab removes query arguments, which were there since the beginning and did not do any harm, at aleast any I could tell. But suddenly this was taken away, without proper alternative. Now you have to hook into their "router" to quickly grab those arguments, before they are gone. This seems silly to me. Why randomly delete query arguments? They are there for a reason and since JupyterLab does not add any of its own, I cannot understand this decision. Simply seems to make it less powerful a tool.

The constant nagging about posting in their community JS-only forum. ("You should post this in the forum.", "Have you seen this post in the forum? links to forum") Why can this community not handle issues in issues, which can be easily found using a search engine. Why hide everything behind a JS-only forum, which one has to create another account for or associate ones Github account with? Whenever anyone gives me a link to the forum, where supposedly the answer to my question is, I keep thinking: "Ahh great, why did you have to hide it in there? If you had documented this in an issue, I would have found it via search engine and the thing would not have wasted my time and neither would I have had to waste yours." -- something along those lines. When I find an issue and its solution, I still post it as Github issue, so that other people can easily find it, without signing up to their forum.

> I haven't really got experience writing extensions in the old Jupyter notebook [...]

I have done that a few years ago, when JupyterLab was still alpha versions. It worked, but the typical JS mistakes plagued me. JupyterLab is of course using TypeScript, which helps a lot with avoiding silly mistakes. However, I do think there is something to what you say about no longer encouraging the quick hack. Some functionality took years to appear in JupyterLab, but was already available for Jupyter Notebook, before JupyterLab took off.

Oh, on the topic of file formats: Quarto also lets you do plaintext notebooks in quite an interesting way, definitely worth checking out: https://quarto.org/docs/computations/python.html

The latest release of nbdev has fully embraced Quarto! It's very awesome, check it out.

I agree re format vs tool complexity. I don't think Jupyter is a particularly difficult format though, its mostly light JSON -- all human-readable.

We realised after working with Jupyter+Git for a while that the pain-points were actually with Jupyter editors (and/or their conventions) rather than the format, because they do things like store user-metadata in the file which pollutes diffs and leads to merge conflicts.

In fact, if Jupyter editors could handle merge conflicted files, we wouldn't need a custom merge driver either.

I have been using jupytext as well and it really makes handling notebooks much easier. I think the decision for Jupyter to save to json was not a good one and they should instead have looked at systems like org mode for inspiration.

I also don't understand what you mean by discipline. Yes you need to make sure that everyone has the jupytext extension installed, but that just becomes part of the needed dev environment. After that the whole experience becomes completely seemless.

Reply on news.ycombinator.com