Jupytext: Jupyter notebooks as Python scripts | Hacker News Reader

Jupytext: Jupyter notebooks as Python scripts

github.com

151 pointsJA7Cal41 comments

This is neat. If you are looking for something a bit simpler: put your code in a python package, install it, and then use %autoreload

You can now develop in your favorite editor, and changes are automatically reloaded.

More detail here: https://blog.godatadriven.com/write-less-terrible-notebook-c...

Well, thank you! No more annoying kernel restarts, that really bugged me a long a time.

This seems neat, but also a bit strange. Are Jupyter notebooks so prevalent that it makes sense to ghave an exporter that makes them into markdown and back again? I mean, I always thought this workflow was neat, but isn't one of the selling points of jupyter the WYSIWYG Interface? And if you like working in code, would it not be easier to create a markdown extension for embedded code (Or does something like this already exist?) and create editor plugins or command line tools to render said document or provide a live preview?

Personally, I totally want that functionality but don't want to mess with running jupyter kernels or using my browser to work in my notebook. So where is my markdown with embedded code WYSIWYG editor that saves into plain text files? ;-)

1 thing. Code review. Even thou notebooks are not meant for production code, it's good to check them into repository to document what you did (let's ignore out of order computation and other horrors), and get feedback on your code / experiment.

Right now, ipython notebook is stored as json, where code is list of json strings (so that you have it line by line).

Additionally, the "WYSIWYG Interface" is horrible in comparison to vim, emacs, idea, pycharm or whatever you are using. What I like about notebooks is being able to present things as experiment: this is the code, and there are the outputs. "WYSIWYG Interface" is good if you want to tweak few things, or if you are starting out.

I think you misunderstood me. My question was not why you would use an external editor (and get all the benefits you provided), but rather, why work back from jupyter to achieve this rather than working forward from simple markdown and allow code blocks to be embedded and run from the editor.

But maybe I am missing some other features than the interface that the ipython kernel provides? I haven't used it much myself, since I found the setup and usage rather complicated, though to be fair, I don't really use python and its ecosystem much.

I would prefer to use editor as yiu describe. but realitybis that jupyter is something lot of people like, find awesone and want to use. this is something that helps someone like me use what other use, but feel less pain. maybe people on future will start migrating out of ui because of stuff like this.

> So where is my markdown with embedded code WYSIWYG editor that saves into plain text files? ;-)

It's here: https://github.com/gregsexton/ob-ipython

Well, yeah. Org mode can do anything, right? If only it wasn't so tightly coupled to emacs. Not a hater btw, I just can't get used to the editor. The concept of an extendable editor sounds really good, but it just seems so complicated and uncomfortable to use...

If you are used to vim maybe spacemacs can help you?

Or a non-spacemacs emacs + evil setup. I jumped ship from Vim but Spacemacs didn't feel quite right so I started from scratch.

As an academic, ipython notebooks are becoming the lingua franca of reproducible research. I don’t like it or them, but they’re excellent for didactic purposes. Extracting the code from the markdown is very helpful for me, as I prefer working directly with it most of the time.

I'm sure notebooks are used a lot for interactive exploration, but I would not use them for reproducible research, so long as they lack things like dependency tracking:

https://twitter.com/joelgrus/status/1033035196428378113

The bar in academia (over-generalizing here) was so low for so long that anything around including your code with your publication still gets placed under the historical title of "reproducibility" - that's probably where the confusion here is arising. I agree with your sentiment, but grandparent probably just meant "a lot of people release their code as notebooks".

It’s a step, but you’re both right. Things that containerize notebooks would count as reproducible, but they’re unwieldy and disgusting. I agree that while better than nothing they’re still not there.

The Python script can be opened with PyCharm

This is going to be a bit pedantic, but for anyone that doesn't realize. Pycharm can already open ipynb files, and it will connect to a running notebook. I don't do it very often, usually just to quickly find my way around some library code.

This project looks great though. I will likely use it, as I tend to use Jupyter to work out code before copying it over to my main project.

In my personal experience, PyCharm really mangles Jupyter Notebooks. Has it gotten to better recently?

I have not experienced much mangeling, but the newest CE edition still doesn't do notebooks very well IMO. to the point where I use jupyter-lab for my notebooks/prototypes and pycahrm for packages.

This is really cool stuff. Editing notebooks in the browser is pretty good, but it can't (yet) match the experience of a well configured (and extended) text editor.

I've been using vscode with the jupyter extension (https://github.com/DonJayamanne/vscodeJupyter) and it's exactly what you describe, experience of a good text editor and the visual notebook workflow.

While working on udacity classes, I found vscode python jupyter plugin very helpful. Having a nice editor makes you so much more productive.

I wrote a little python pip module to help me convert between .ipynb and .py.

https://github.com/nojvek/vscode-ipynb-py-converter

Made a world of difference.

Neat, I'll give that a shot. Does it handle all kernels?

It doesn't appear to. Helpfully, Hydrogen [0] for Atom was designed to be entirely language agnostic.

[0]: https://github.com/nteract/hydrogen

If you're worried about diffs, check out https://github.com/jupyter/nbdime

That's true but ideally you would like the native format to be easily diffable (e.g., plain text) and not require special tools.

Could this be the answer to "I don't like notebooks" [0]? Or does calling scripts from within a notebook still do horrible things to the managed state?

[0] https://news.ycombinator.com/item?id=17856700

I've never thought this criticism was that relevant. Notebooks have been incredibly successful, and if it's possible to call your code in the wrong order and create a mess, it's probably a problem with your code not your coding environment. The more functional your code, the less of a problem jupyter notebooks are. They encourage better code not worse imo.

The "managed state" is important, though. For example I deal with cases where I'm doing some big expensive operations upfront (eg, processing a bunch of data coming in over a network share— ROS bag files, which are also bzipped, adding more upfront processing cost), and I'm interesting in tweaking the plots and analysis coming out the other end without having to re-run the processing each time.

Even as a relatively experienced developer, I've found it hard to reason about what exactly it is that Jupyter is doing under the hood; basically things will break in weird ways, and rerunning everything will mysteriously fix it, or I have to use hacks to force it to re-import certain modules in order to get things the way I expect.

Basically, it ends up being easier to just manage this myself, like doing the processing and pickling the result, then using other, standalone notebooks to load and render the pickle. But this shouldn't be necessary.

That sounds really weird. The only time I noticed such weirdness was when a colleague and me were simultaneously editing a notebook on Google Colab. The code was shared but not the state, super weird...

I think this partially matches the parts of the criticism about writing a book, and how the notebook is unsuitable - mainly thinking along similar lines to writing software tutorials pages that are fully testable.

After seeing that post I had a look around for something a little like this and found Pweave http://mpastell.com/pweave/ which seems to approach the problem from the inverse direction direction, but I think I'd rather have something like this where the code is marked up and runnable as normal code but transformable into formatted documentation.

This seems like a reasonable match, but it looks like it's explicitly tied to notebooks and I'm not completely sold on the overloading of plain comments for markup purposes. Still, probably a good step.

This is fantastic!

Now, they just need to adapt jupyter itself so that it reads the python files directly, dispensing with the need of the silly ipynb!

I think that’s called IPython. But I agree with your sentiment.

I was not being ironic. Notebooks are lovely as an output format for showing your research and allowing others to fiddle with your code. However, for people who do not want to edit inside a web browser window, jupytext is a godsend!

At that point you should put most of your work in a separate file, and just import the stuff you need for your demo.

In such a use case, the whole point is showing your work in a 'literate programming' manner i.e. having the appropriate code that's being run be right there between the descriptive text, instead of just running do_something() that you imported from somewhere.

I do not want to hide anything, my only reason to use notebooks is to put all the computations inside them.

Nice! similar to https://github.com/minodes/pynb , that converts Jupyter Notebooks to Python scripts you can run and back, preserving Markdown cells.

Can someone provide me a usecase for this when compared to executing jupyter straight into VSCode/Atom with vscodeJupyter/Hydrogen?

The idea is that you can edit your notebooks with your favorite text editor, use git for version tracking them, and no need to mess with a web browser at all.

In the medium term, it's likely that Hydrogen will be able to read and write .ipynb files. It can currently export .ipynb files but has some bugs [0].

[0]: https://github.com/nteract/hydrogen/issues/1296

I have written https://github.com/azag0/knitj to solve a similar use case, but with an alternative interactive browser interface

I really like the markdown conversion! This should make it easy to import ipynb notebooks into markdown based documentations.

Reply on news.ycombinator.com