Comment by ptero | Hacker News Reader

Comment by ptero | Hacker News Reader

Parent Full thread

For a while I viewed Jupyter as a toy that is neither here nor there (sitting between the chairs of development and explanation, briefing or visualization and not doing either job great). But about 2 years ago when I changed jobs into a "Jupyter heavy" environment I was forced to learn it and have grown to really like it.

I primarily use Jupyter for prototyping: trying ideas, plotting results and sharing notebooks for others to improve on (or punch holes in). Once a piece clearly shows promise I move it into a module. While this usually means a significant rewrite I actually see it as a benefit: I can be messy in original prototyping plus a rewrite after experimentation often leads to a better code with a small time investment.

Beyond the minimal self-discipline of actually moving code into modules I still have two three-character friction points: vim and git. Pointers on addressing those appreciated!

I would love, love, love a tight vim and Jupyter integration to be able to switch, easily and frequently, between editing a set of cells in vim and in Jupyter with solid sync between them. I am perfectly OK with vim ignoring the output. And I would love to have a git mode that only checks in the changes that cause actual differences in the python code; not timestamps or the output.

You can comfortably edit the whole thing as Markdown in VSCode (in vim mode, if you'd prefer) - and still have executable cells and nice pdf/html rendering. https://quarto.org/docs/tools/vscode.html

EDIT: actually, you can do it directly in neovim - https://quarto.org/docs/tools/neovim.html

This is my preferred way to work as well. Some techniques to address what you are missing:

1. Only checking in semantic differences (not output, timestamps, etc.):

Use the `jupytext`-extension [0] to seamlessly pair you notebook with an lightweight markup version of the input only (which can be used to generate full notebook).

2. Being able to switch between text editor and notebook interface

As other have mentioned, there are integrations for multiple editors.

Another approach is to move the central code out to a python module which you edit in a text editor, and then use the `%autoreload` magic [1] to reimport that module whenever you execute a cell in the notebook.

[0]: https://jupytext.readthedocs.io/en/latest/install.html [1]: https://ipython.readthedocs.io/en/stable/config/extensions/a...

This seems like a nice approach. Mine is similar. I refactor often, by moving stuff from a lower level of discipline to a higher one:

Enclose scripts in functions within the notebook, which minimizes the clutter of hidden state. I also have a habit of not walking away from a notebook without doing a "restart kernel and run all cells" to make sure the notebook works. I'm not dealing with giant data sets, so this doesn't cost me much.

Frequently used functions go into .py files, using auto-reload to keep things synchronized while I'm working on them.

Mature .py files that I might want to re-use in different projects get turned into pip-installable packages. The notebooks become informal tests of the packages.

I've never used venv, and never encountered dependency version problems. Some of the dependency horror stories may be obsolete due to the maturation of the big packages such as numpy and matplotlib.

This is a plugin that makes browser text area elements into nvim buffers.

https://github.com/glacambre/firenvim

Haven't tested it in combination with Jupyter but I imagine it should work

Are there some usecases where conventional IDE isn't up to the job as the notebooks? I can imagine this to be so for data analysis where pre-loading heavy datasets saves time. Anything else?

For me its the matter of actually liking The notebook environment rather than overcoming the shortconings of IDEs.

The notebook environment is excellent for data analysis and exploration. It matches my workflow of manhandling data as I reach for understanding.

Once I figure things out of course, either I'm done with the notebook and can copy a few plots out for inclusion in a powerpoint, or I'm done with the notebook and extract a pile of functions into a utility script or package.

Either way, it has served its purpose for rapid prototyping and one-off analysis.

jupytext will solve the git problems for you. Pair with a percent script, ignore the ipynb files.

jupytext is great. It even allows you to use only .py files directly as notebooks, but I recommend "pair with ipynb" and version controlling the .py file. The ipynb acts like a cache of the cell outputs between invocations of jupyterlab, which is handy too.

I always wanted a Jupyter like environment, but the one that would natively support and output .py files

jupyter notebook as IDE, but with .py files instead of .ipynb

This is exactly what jupytext gives you. The "pairing" is optional (and sometimed confusing). You can just work with .py files as notebooks and never ever see a .ipynb on your disk again.

The notebook .py files are just regular python files with comments that can be edited at hand wih any text editor. Thus you can easily collaborate with your local graybeards that will dislike editing text on their web browsers.

What I miss is a notebook launcher button that creates a .py notebook. Don't understand why we don't have that (i could not myself find a way to configure and add it.)

I'm not sure why you think it's viable to collaborate with others who don't use jupyter? I mean on any software project, if we don't agree how to compile or run a project, then we usually can't collaborate IMO. Changes become nonsensical (breaking the one or other mode that is not tested by the author of the change.)

I think collaborating with .py files is easier over git.

I could not find a way how to check-in .ipynb file and let reviewer review code only, and ignore the metadata part

if jupytext will allow me to work in jupyter, check in file to git and create MR and let other people easily see, review and comment my code that would be great

Great, then I'd recommend using jupyterlab 3.6 with jupytext right now

There are lots of different versions of this. One of these approaches may work for you https://alpha2phi.medium.com/jupyter-notebook-vim-neovim-c2d...

Reply on news.ycombinator.com