Notebooks as reusable Python programs
marimo.io
marimo.io
You can work your way around it by storing the results in a separate file(s), and writing the boiler plate to let the reader load the results. Or they let you export to ipynb - which is still sharing two files.
Presumably the reason for this decision is making git diffs short. But to me the solution is to fix git diff to operate on JSON nicely, rather than changing the entire notebook format.
Arguably this is a good thing. You shouldn't distribute things you can't prove have the same results and one way to do that is to require others to run the same computations.
Most learners do not need to fact check the instructor, but do want to see the operations which were run. Those that are curious can run/edit the notebook themselves.
Edit: The JupyterBook ecosystem (https://executablebooks.org/en/latest/gallery/) as an example of what is possible with stored plots/calculations. Most learners are just going to follow along with the material, but being able to optionally play with the data is the super power of the platform with minimal friction.
Why not? Can you expand on this because I don't see why this is not a good thing.
Besides if you distribute your code alongside your output, aren't you providing that proof anyway? People can run your code and see they are getting the same result.
I have looked at marimo several times, and while it's great for interactive computing, and it has a fantastic team, it's not a replacement for notebooks and I find their use of the term "notebook" confusing. As a scientist, I don't understand what use case they are exploring, but I do know it's not the use case where Jupyter was created and it's not my current use case for Jupyter on teams I work with.
small edits to code yield enormous Git diffs;
code is copy-pasted across notebooks, instead of reused;
magic commands limit the portability of notebook code;
logic that would be useful as a script or library gets thrown away;
logic that should be tested almost never is.
The `export` command can generate a rendered artifact if that's what you're after but there is also another avenue here, have you seen the caching feature? The one that caches to disk and persists?
https://docs.marimo.io/guides/expensive_notebooks/?h=cache#d...
This can automatically store the output of expensive functions, keeping the previous state of the cells in mind. If a re-compute ever needs to happen it will just load it straight from cache.
Another option is to run in lazy mode, documented here:
https://docs.marimo.io/guides/expensive_notebooks/?h=cache#l...
This will prevent the notebook from rerunning cells by accident.
We're thinking about adding features that would make marimo great for running long running batch work but there's not a whole lot I can share about it yet. If you have specific thoughts or concerns though, feel free to join our discord!
If I understand it correctly, `@mo.persistent_cache(name="my_cache")` creates a binary file `my_cache` that I should commit as well if I don't want others to repeat the computation?
This kinda solves the problem, except for having two files per notebook, and that marimo notebooks are no longer viewable with output on github directly.
We plan to add more stores like Redis, S3-bucket, or an external server, since you may not always want to commit this file, but like you said want others to avoid the computation.
Is that really what's wanted? If there's some cell that I need to run twice for some reason that I tried debugging but wasn't able to figure out why, or for debug just run cells 1-5 in order but on specific other (prod) systems, skip cell 4 and run 3 before 2. Now, arguably well written software would handle those things automatically, but we're not talking about battle hardened software that's had an SRE team vigorously refactor it until it's been proven suitable for such purpose by having been on call for it for months off not years. We're talking about notebooks, which have their time and place, but the entire point, I would argue, is to make it easier run notebooks in production without the added overhead of said SRE team. And in that world, the reality is the PhD is gonna have some things they know they should fix, but it's easier to just comment out the unneeded cells on the prod and hit run all, so shouldn't the tools better support that use case (by saving what was actually run and offering to rerun that) over caching based on input string and hoping for the best?
marimo lets you automatically snapshot outputs in an auxiliary file while you work. We also have a persistent cache that lets you pick up where you left off — saving not just outputs but also computed data.
Many of our users do very heavy computation.
This is not quite true. Outputs are not stored...in the Python file*. marimo does store outputs in the `/__marimo__` folder with settings enabled.
> writing the boiler plate to let the reader load the results.
Therea are some primitives to do this for you, such as mo.persistent_cache. This can be an annotation or 'with' block. It intelligently knows when either the source code or inputs change.
The plan is to take this one step further than storing just the output. Because marimo knows the dependencies of each cell, in a future version, it will store each output AND let you know which are stale based on code changes. This is being built by Dylan (from the blog) and inspired by Nix.
0: https://code.visualstudio.com/docs/python/jupyter-support-py
[0] https://docs.spyder-ide.org/current/panes/editor.html#code-c...
# %% [markdown]
# Another Markdown cell
# %%
# This is a code cell
class A():
def one():
return 1
# %% Optional title [cell type] key="value"
```{code-cell} LANGUAGE
:key: value
CODE TO BE EXECUTED
```
---
kernelspec:
name: javascript
display_name: JavaScript
---
# Another markdown cell
```{code-cell} javascript
// This is a code cell
console.log("hello javascript kernel");
```
The vim plugin I was talking about was vim-slime [0], which seems to date from 2007 and does have regions delimited with #%%.
Slime comes from Emacs originally, but I could not find if the original Emacs slime has regions.
Matlab also has those, which they call code sections [1]. Hard to find when they were introduced. Maybe 2021, but I suspect older.
None of those stores gge output of the command.
[0] https://vimawesome.com/plugin/vim-slime
[1] https://www.mathworks.com/help/matlab/matlab_prog/create-and...
# %%
"^# %%"
Org-mode supports code blocks: https://orgmode.org/manual/Structure-of-Code-Blocks.html :
#+BEGIN_SRC <language> and
#+END_SRC.
Notebook interface; Markdown + MathTeX: https://en.wikipedia.org/wiki/Notebook_interface ;
$ \delta_{2 3 4} = 5 $
$$ \Delta_\text{3 4 5} = 56 $$whether or not it's better for you depends on your use case for notebooks — i use them mostly for prototyping and exploratory data analysis so separating the code from the output might be more convenient for me than for you
I wrote a small vscode extension that builds upon this to automatically infer code blocks via indentation, so that you don't have to select them manually: [0]
You can have a plaintext file which is also the program which is also the documentation/notebook/website/etc. It's extremely powerful, and is a compelling example of literate programming.
Decent overview here: https://www.johndcook.com/blog/2022/08/02/org-babel-vs-jupyt...
[edit: better link]
You can do most work in org instead of editing programs directly, and it'll save time and produce good documentation. Can't recommend it enough.
This approach has the recently developed benefit of being really great as context stores for interacting with LLMs.
One (not a great) workflow I have, is that I use notebooks as quicks UIs to visualize some results. 1. Run a simulation that outputs results to some file 2. Load results in a notebook and do some quick processing + visualization
Very often, I want to compare quickly between 2 different runs and end up copying down the cell with visualization, then just re-run the data load + processing + visualization and compare them.
My understanding is that this would not be possible with marimo, since it will re-run automatically the cell with my previous data right ?
choice = mo.ui.dropdown(['train', 'split')
data = load(choice.value)
processed = process(data)
visualize(processed)
This way, you can toggle between just more than two if needed. If you need to see both at once, you'd want to refactor the processing and visualizing step into functions, and then just duplicate the finals cell(s).
marimo has a multi-column mode, so you can view them side-by-side
I have had a similar situation and my "hack" for this was to start the same notebook twice and have two tabs open. This worked for something things ...
Other times I just bit the bullet and made two forms on two variables so that everything would fit in a single notebook. By having two variables that contain all the inputs you can make sure that only the cells update that need to update. It takes a bit more effort but makes a lot of sense for some apps that you want to share with a colleague.
If you share more details about your setup I might be able to give better advice/think along more.
This is such a small UX thing but it’s so damn important. The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. It might add the smallest amount of friction to the UX, but it’s so damn important.
Also the choice of json as the file format is just plain wrong. Why the project hasn’t just abandoned that entirely and done a json-#python back and forth when writing to file is beyond me. There are extensions that do this, but that’s a really clunky interface, and while I can set it up for myself it’s difficult to force upon others in a corporate environment.
Great to see someone is taking the seemingly small things up, because they mean a world of difference to the overall ecosystem.
How you start the marimo notebook, via
`marimo edit must-give-name-to-this-file.py`
is indeed one of my teeny by favourite features of it. When you start a new notebook you're kind of forced to name it immediately.
The even simpler fix is to just not name them until the user does. That's the way other programs work. If you create a new document in a word processor, it will say "Untitled" at the top of the window, but it doesn't create a file called untitled.doc on disk until you do "Save as" and choose a filename. It has always irritated me that Jupyter insists on having an on-disk file right from the beginning.
It would also be great to have a decent search across notebooks in a project so I could quickly find an old function.
This is false. Org-mode has had this functionality for over two decades.
The team is responsive and they care about getting it right. Having a sane file format for serializing notebooks is an example of this. They are thinking about core problems. They are also building in the open.
The core jupyter team is very unresponsive and unfocused. When you have a bug, you need to figure out which one of many many interelated projects caused the bug, issues go weeks without a response. It's a mess.
Then there are the proprietary notebook like environments. VSCode notebooks, and google colab in particular. They frequently rely on opaque undocumented APIs and are also very unresponsive.
Happy to hear it! Got a link to your widget? I am assuming it's an anywidget?
Me, I’d have support in the parser for a language structure or magic comment that points the cell boundaries. I would make dynamic execution of code a first party feature with history tracking and export of all the code sent into the runtime through this. Thus what the runtime saw happen could be committed over what user thought they did. Also, a better, notebook aware evaluation with extension hooks for different usage methods (interactive, scripting, testing).
I have no solution to ipynb JSON problem. I do think it is bad that our seemingly only solution for version control can manage only simple text, and users of all the other format have to adapt or suffer.
Being able to build model/simulations easily and being able to share them with others, who can then even interact with the results, as truly motivated me to try more stuff and build more. I've been deploying more and more of these apps as PoCs to prospects and people really like them as well.
Big thanks to the team!
On running notebooks as scripts: I can’t find in the docs what happens if you have plotting and other notebook oriented code? Like I’m using pygwalker to explore data through transformation steps, and end with saving to csv. If I just run the notebook as a script, is all of the plotting automatically skipped?
It depends a bit on how the notebook is written. There is `mo.app_meta()` that allows you to detect how the notebook is running. This can be in "app mode", "edit mode" or in "script mode".
https://docs.marimo.io/api/app/?h=meta#marimo.app_meta
Effectively this could allow you to do things like "only run this bit when not in script mode" if you want to skip things.
Alternatively you can also run the notebook via the `marimo export` command if you care about the charts and want to have a rendered notebook as an artifact.
Gotta ask out of curiosity, anything you can share about your cursor/marimo workflow? Are you using the llm tools from within marimo or outside of it?
However, I'm not sure what the use case is for Marimo. I see Jupyter notebooks being used in two primary use cases: 1) prototyping new code and interactions with services and databases and datasets, as a record of the REPL used to understand something, with interactive notes and plots and pasted in images from docs, etc. 2) a record of how a calculation was performed, experimental data analyzed, and in a permanent artifact that others can look up later. For both of these, outputs and markdown/image cells are just as important as the code cells. These are both "write once" types of things where changes in git are rare, and ideally would never happen.
With Marimo, can I check the outputs directory into version control in a reasonable way and have it stored for posterity? Is that .ipynb?
Is there a way to convert a stored .ipynb checkpoint back into the marimo format?
And why does a small .ipynb change lead to many lines of change in the git diff? It's because the outputs changed. Deciding to not store outputs in version control and counting it as a win for pretty git diffs is saying "this core feature of .ipynb should be ignored because it's inconvenient". I'd much rather educate people about turning on GitHub's visual Jupyter diff rather than switch to an environment where I can no longer store outputs inline.
Similarly, being able to import one cell into a different notebook seems like the wrong direction to solve the problem of "it's time to turn the prototype notebook into a reusable module." If it's time to reuse a cell, it's time to make a cleaned-up Python code module file, not have the code interspersed with all the rest of the stuff.
I'd like to learn more about the use cases where Marimo is useful. As a scientist, it's not useful to me. I don't care about smaller git diffs on a notebook, in fact if a notebook is getting changed and re-checked into version control then a big awkward diff is not a problem and probably a feature, because notebooks should not be getting changed. They are a notebook something that you write in once and it's done!
You could argue this is a user issue, and being sloppy with cell state is a me problem- not a notebook problem- but marimo prevents this in the first place. Just once is already a wasted day's work. Even if one is especially diligent in Jupyter, they do not have the guarantees marimo provides. Unless you are pressing "restart and run all" in a Jupyter kernel every time - you are accumulating state.
I got started with marimo and helped build out the caching because I wanted to open up a notebook from scratch and automatically restart at the point I was at- without having to worry whether my cell order is correct or not. I wanted my code to automatically rerun update outputs if I changed something like a filtering or transform step to my data. I also wanted to go back to previous states easily without having to rerun long computations or deal with data dump versioning.
marimo's caching makes this all automatic.
The plan is to bring caching to an opt-in cell basis so this happens behind the scenes, with some ideas for handling side-effects like randomness and other "non-pure" cases.
But to answer your points, my use case has been specifically scientific computing. Yes for 1) prototyping- but also knowing my prototype does not hold some hidden state or bug. I have shown my advisor results, gone to deploy and been unable to replicate them at scale because of some hidden bug in my notebook execution. To me this is unacceptable. but also 2) documentation, notes, and report sharing. Not saving the outputs directly with the code isn't a blocker here, because caching automatically restores my state, and I can export to a more suitable format like a PDF or HTML file if I need to share it. Saving generated binaries in source for any other setup would be considered poor form- I don't understand why Jupyter gets a pass here. "Write once" is a google doc, notebooks should be "run many times", i.e. reproducible and sharable.
Hopefully, we get to address some of the other pain points with existing infrastructure; like auto-scaling on HPCs (easy for marimo, it's already a DAG); integrating code more directly into papers (we have mkdocs, and experimental quarto extension exists); and built-in environments (no more `pip install -R requirements.txt` on a paper project and hoping the authors remembered to freeze and pinned versions)
---
Hope my tone doesn't come off too contentious- I've just been burned by Jupyter and I'm excited because I feel like marimo addresses its major issues.
I think your use case makes a ton of sense for Marimo, iterative modeling and mucking around. Most of my modeling has been less like that, and goes straight to a single computation with the result. That is a significant workflow difference.
The jupyter lab web interface has many many UX problems that make it extremely difficult, as does the whole kernel launching infrastructure. Having a desktop app, or another way to launch kernels in a daemon like mode without keeping a terminal open, could save a ton of headaches.
Marimo is great in that it accomodates pure functional programming fantastically for notebooks. Unfortunately too much of the Python API ecosystem does not, but that doesn't mean that it couldn't in the future.
> With Marimo, can I check the outputs directory into version control in a reasonable way and have it stored for posterity? Is that .ipynb?
You can snapshot to ipynb automatically while a marimo notebook is running, or after the fact. You can also snapshot as HTML, which marimo can hydrate back into a notebook.
> Is there a way to convert a stored .ipynb checkpoint back into the marimo format?
Yes, `marimo convert notebook.ipynb -o notebook.py`.
marimo was originally commissioned by, and designed in collaboration with, scientists at Stanford's SLAC National Laboratory. These scientists were heavy users of Jupyter, but decided they needed something like marimo to solve two main problems they encountered with Jupyter notebooks: computational reproducibility, and publishing interactive science communication on the web.
We have co-authored an article with these scientists, explaining how these problems and how marimo solves, which you can read here: https://marimo.io/blog/slac-marimo.
From our article:
> In 2019, a study from New York University and Federal Fluminense University found that of the 863,878 Jupyter notebooks on GitHub with valid execution orders, only 24% could be re-run, and just 4% reproduced the same results.
Many scientists (these scientists and myself) use notebooks as a critical part of the scientific process. Jupyter notebooks are notorious for being plagued with reproducibility issues due to hidden state, to the extent that the authors of Jupyter wrote a paper titled "Ten simple rules for reproducible research in Jupyter notebooks" [1]. In my mind, these rules are not simple nor convenient. One is to periodically restart the kernel to make sure you don't accumulate hidden state — untenable for expensive notebook. Another rule is in fact to use version control (Git):
> Version control is a critical adjunct to notebook use, because the interactive nature of notebooks makes it easy to accidentally change or delete important content. Furthermore, since notebooks contain code, and code inevitably contains bugs, being able to determine the history of when a given bug you have discovered was introduced to the code vs when it was fixed – and thus what analyses it may have affected – is a key capability in scientific computation.
In my own PhD, I used Jupyter notebooks extensively to see my data while I worked on it, and to produce figures for papers [2]; sometimes, these notebooks took a very long time to run. I have a background in software engineering, so I was able to make my notebooks more or less reproducible, though I wasted many hours debugging inconsistencies due to hidden state (one example: delete a cell, but its variable are still in memory!) My co-authors, however, often did not have a background in software. They also used Jupyter notebooks. But when I tried to run their notebooks, either they didn't work at all (packages and computational environment not properly documented), or when running I got different results than the ones serialized in the notebook (notebook author ran cells out of order, or ran side-effecting cells multiple times, etc -- hidden state). This was a huge problem and really a non-starter for computational science.
Yes, you can get Jupyter/Ipykernel to work for you if you try hard enough, if you are experienced enough, and if you are willing to eat enough pain (sometimes literally -- I know of someone who got an incorrect tattoo due to hidden state in a Jupyter notebook that rendered a design for their tattoo; the notebook author forgot to "restart and run all" after making a change). But the imperative nature of the REPL makes it a fundamentally error-prone experience. Our philosophy with marimo is that notebooks should be reproducible by default.
Hope this helps. It's still possible that marimo is not for you, but at least for scientists who use notebooks to conduct research, or want to share interactive articles on the web without paying for compute, I do believe it has something compelling to offer over traditional notebooks like Jupyter.
[1] https://arxiv.org/pdf/1810.08055 [2] https://web.stanford.edu/~boyd/papers/min_dist_emb.html
RMarkdown is essentially Markdown with executable code blocks. While it comes from an R background, code blocks can be written in any language (and you can mix multiple languages).
The biggest difference (and, I would say, advantage) is that it separates code from output, making it work well with version control.
The reactive component is great for exploring data in an easy way, with sliders and buttons. Also for simulations when you want to try a bunch of different parameters.
marimo edit notebook.md