Lessons learned reinventing the Python notebook | Hacker News Reader

Lessons learned reinventing the Python notebook

marimo.io

106 pointsmpweiher36 comments

I find it so weird to see Jupyter being the "traditional" option in this discussion.

Kudos for the shout out to excel being the main "notebook" most people have experience with.

I question the choice of Python as the format. Agreed that Json is terrible. But Python will have similar problems. And a notebook is a marked up document. Using a markup language is a natural choice. Delimit areas of particular syntax and you are off to the races. (Yes, I am fond of org mode.)

We're adding support markdown with fenced code blocks, which maybe gets you closer to "delimiting areas of particular syntax": https://github.com/marimo-team/marimo/pull/1332.

Python is nice because it enables composition and reuse as vanilla scripts. But there are definitely tradeoffs.

Yes, fenced blocks is what I meant. Recreating parts of xml and cdata seems the common end goal. I think we have enough modern processing that you can do looser grammars.

Composition is an odd goal to me. Most notebook usage is exploratory. As such, taking effort to reuse notebooks feels counter to the benefits of doing it again that you can get.

Though, I'm a fan of seeing different workflows. And excited to be shown places I'm wrong! Kudos!

How about lessons learned from other reactive notebook environments that have been around for a while? If you are 'reinventing' something, I wonder why no mentions are made of the paths that have been laid before (Observable, Quarto, Enso, etc.).

Their GitHub does ;-)

I enjoyed the article. We are doing a more genAI-oriented notebook env variant for louie.ai aimed at database/API users preferring natural-language-first vs typical Jupyter/observable/hex/etc python wrappers, and for their automatic reactivity portion, a lot of similarity here with the road we went down. I appreciate seeing some of that reasoning laid out. Especially neat is the use of static analysis of global assignments vs the typical input/output variable syntactic convention. Few go so fine-grained (enso, frp, ...).

My personal experiences with automatic reactivity (OSS'ing the first web frp system, and many after) and last 10 years as a heavy Jupyter notebooks user with many enterprise/gov/tech DS+operational teams actually steered us down a different path. We separate out the auto-reactivity to only auto-trigger when switched into a 'live' dashboarding view. By default, when in the authoring notebook view, we stick to Jupyter-style manually triggered run / runall. When authoring, the notebook accidently running a bunch of database or OpenAI or bigquery calls when you didn't intend can be... problematic. The underlying cell-based reactivity model we went with is largely the same, but when an analyst is dealing with big database calls and returns, a world of a difference in UX.

Fwiw, if anyone uses Splunk/databricks/OpenSearch/etc and curious about a team genAI-first learning system with auto-viz-reactivity, would love to chat!

Pluto.jl is our biggest inspiration. We also took some inspiration from streamlit, and of course a lot of inspiration from the IPython kernel and Jupyter. We didn't look closely at Observable (though Fons cites Observable as his biggest inspiration), and didn't look at Quarto or Enso at all.

As others have mentioned, we're very open about our inspiration and document it in many places, such as on GitHub, though you're totally right that we could have mentioned it here too.

We could fill another blog post talking about inspiration and lessons learned from other projects — perhaps we will!

No mention of Julia's Pluto seems like a real miss. I am Python day-to-day, but the Pluto demos I have seen are very attractive.

I think Pluto also lets you reassign a variable.

I'm a marimo author. Pluto is amazing, and definitely one of our biggest inspirations. As others have mentioned, we've given shout outs to Pluto.JL in most of our copy; oversight to not include it here.

i spent time w pluto. it's not at parity

I'm one of the authors of marimo; happy to answer any questions.

Also see the previous discussion here: https://news.ycombinator.com/item?id=38971966

as a daily jupyter user, i like this .py DAG approach a lot - more so for visualizing complex workflows than addressing a potentially corrupt hidden state.

there isn't much *data science functionality* integrated into jupyter core apart from displaying static dataframes (nor is their internal desire to add it), and jupyter *extensions/widgets* should really be built with dash/streamlit, so i'd encourage you to make an ecosystem.

having spent a lot of time with airflow recently, it's designed for scheduling recurring jobs, not *batch processing.* meanwhile dagster is not meant for computationally intensive/ long-running jobs and the dataset-centric approach is offputting

overengineering = since you made the cells functions already, it would be cool if you could make multiprocessing/multithreading an option in the cell UI and help keep track of multi-state variables/execution

happy to give feedback/ user interview = linkedin.com/in/laynesadler/

why the downvote? debate me. it's saturday and i'm working in jupyterlab

In principle every notebook is a DAG, since the cells are evaluated in a certain order. You could even infer the DAG from Jupyter notebooks, at least when the execution number is stored (the In[...] part).

There is a lot of value in imposing restrictions on the cells (like no reallocation of global variables) to be able to infer the DAG more directly like they do here (or Pluto.jl does). It is simpler, it can serve a dual purpose by being a normal runnable python/Julia/whatever program. Another advantage that I didn't see mentioned in TFA is parallelism. There is some interesting potential there I think.

> Another advantage that I didn't see mentioned in TFA is parallelism

That's something we're thinking about and designing:

https://github.com/marimo-team/marimo/issues/1103

Feel free to join the conversation there if you have thoughts!

A lot of improvements!

Relevant to my 2020 article: What’s wrong with computational notebooks?

https://austinhenley.com/blog/notebookpainpoints.html

Yes! I remember reading your article very closely a few years ago, before starting work on marimo. Thanks for writing it and sharing it here.

Other things we looked at closely: Joel Grus' (in)famous talk on notebooks, and obviously Pluto.jl.

How well does this work in practice? I am skeptical about the power of static analysis with a language like Python. Many times, the hidden state in Jupyter lives in another module, or somewhere deep in an object hierarchy.

If you’re going to reinvent the Python notebook for reproducibility, I wonder why not go further, and fully snapshot the program state?

I really want this to work, because hidden state in Jupyter notebooks bites me frequently, but the post and website haven’t convinced me this is a robust approach - maybe I’m missing something?

> marimo makes it seamless to go from prototype to production.

No, it doesn’t.

In fact, no tool; not retool, not any of no-code solutions, free or paid, lets you take a hacked up prototype and seamlessly put in into production.

In fact, even claiming that it does makes me question the other (superficially quite laudable) things it claims to do; eg. plain python files.

Why do people go down these rabbit holes?

You can’t run prototype notebooks for your production workloads.

It’s not because it’s fundamentally impossible to use a notebook as a platform for robust production code.

…it’s because prototyping means quick hacks, quick dirty unmaintainable hacks.

> Run your notebook as a web app, with Python code hidden and uneditable

Why? Stop. No. Don’t make your prototype hacked up notebook that has probably got direct system access public.

You at least are password protecting this right? No? (https://github.com/marimo-team/marimo/issues/395)

What part of don’t accept every damn idea as a feature did you not take to heart?

-_-

There are a lot of ideas in this project. A LOT of ideas. Some of them are good. Some of them are fully implemented.

production != public, in most cases.

If you are making things public, you likely care about more than just a password, which you'll want to push into an auth gateway with oauth support, authZ, rate-limiting, etc.

.py is undeniably more widely compatible in production envs than .ipynb

token/password protection is in the works; just designed it with my co-maintainer yesterday.

Yours is the fundamentalist view and it always has its place in the debate. But it is not necessarily what the organization needs.

There are benefits of running notebooks in production rather than reimplementing things in another environment.

1. The prototype is tested and verified. And I don't mean tested as in 50 unit tests and integration tests. But tested in the way that a human actually looked at the result and was satisfied. Reimplementing means that you will start from scratch.

2. That the production env is the same as the analysis env means that it is trivial to go back and debug issues.

3. Sometimes maybe what you need is a fast solution. Not sending stuff into the scrum loop of a separate engineering team and pray it makes it into a sprint not so far away.

TL;DR get shit done.

If the target is C++ you can use my library https://github.com/xlladdins/xll. It makes it easy to call C++ from Excel.

I am in the market for an alternative to Databricks' notebooks. Essentially, I want to use spark-connect, but also interact with delta tables directly via polars and duckdb (while still leveraging Ibis).

Support for Observable tier plots would be amazing.

similar to how you parse the cell's input variables, it would be cool if you parsed the files being read (e.g. `pd.read_parquet` and `file.open`) or databases/sources. you could show metadata about the sources next to it.

this UI setting could be turned off if every cell i/o from the same file

the only ergonomic solution to me is to maintain a converted script of notebooks and using it for diff tracking, productionizing, etc. through a simple abstraction.

whoo! that you could jump back up and use variables from below has been an annoyance of mine

in the words of Fernando Perez .. "don't fork the notebook"

“I had 3 desiderata for it”

Ummm why use such a fancy word for requirement ? Put me off reading the whole article.

Reply on news.ycombinator.com