Then I started mentoring a junior staff member who worked on another project. The lead of that project was a physicist who wrote primarily in Jupyter notebooks. Like, thousands and thousands of lines of code. This junior staff member spent like 90% of her time confused and copying-and-pasting between notebooks. She had no idea what a virtual environment was and her go-to solution for solving import errors was to nuke and re-download the repo and re-run the setup script, or create a new notebook from scratch containing all the prerequisite user functions.
Was top 10 most horrifying things I’ve ever seen. I advised her immediately to stop using notebooks for development and sent her a few Python tutorials. Luckily though she just left the project and got on a better one instead.
Good SWE practices are 100% not taught in school.
A REPL, by its name, is a very narrow version of the broader paradigm of interactive computer programming environments. But Notebooks are not REPLs, unless you use REPL to mean "interactive programming environment" and not REPL. Notebooks are much broader than a REPL! In a notebook, you can go back and edit and run individual lines of the notebook without re-running the whole notebook from the start and without re-computing everything that depends on what you just edited. Behavior like this makes it super hard to track the actual state, and super easy to lose track of how things are how they are. That's pretty terrible!
The parent article links this great talk that goes into more detail than the parent post and is much easier to understand: https://www.youtube.com/watch?v=7jiPeIFXb6U
Isn't this a standard on REPLs as well? You can select the code you wish to run, and press Ctrl+Enter or what ever. I must admit, I've programmed Python for about 10 years in Spyder and VS Code now, but I haven't used notebooks at any point. Just either ad-hoc scripts or actual source files.
My definition of a "notebook" is an ad-hoc script, split into individual "cells" which are typically run as a whole. On my workflow, I just select the code I wish to run. Sometimes it is one expression, one line, 100 lines or 1000 lines depending what I've changed on the script.
Not usually, no. Type `python` at the command prompt - what you get is a REPL. Type `clisp` at the command prompt, or `wish`, or `psql`, or `perl` or even `bash` - those are al REPLs.
Very different to a program that presents an editor, and then lets the user selectively choose which lines/expressions in that editor to run next. For example, type `emacs somefile.sql` in the command prompt. The application that opens is most definitely not a READ-EVAL-PRINT-LOOP.
For the same reason that adding (working) wings to a car makes it not a car anymore.[1]
I mean, to my mind, when something is satisfying a different primary use-case, then that thing is a different thing.
I'm sure there's some fuzziness in the distinction between "This is a REPL/car and this is a Notebook/plane".
Usually it's very easy to see the distinction - the REPL is waiting for the next command and the next command only while the notebook takes whatever input you give it, determines whether it got a command or content, and reacts appropriately.
[1] Tons of examples, TBH. I don't refer to my computer as my calculator, even though the computer does everything a fancy calculator can do. People don't call motorcycles 'bicycles', even though the motorcycle can go anywhere that a legal bicycle can go. More telling is how people don't call their computer monitor 'TV' and don't call the TV a 'Monitor' even when the same actual item is used for both (i.e. I repurposed an old monitor as a small-screen netflix-box, and now an item that used to be called 'monitor' by wife and kids is called 'TV' by wife and kids).
Jupyter Notebook absolutely is a REPL, see my sibling comment above for the WP link describing it as such. It waits for input, then evals the input, the prints the return value, and then loops.
“A Jupyter Notebook application is a browser-based REPL containing an ordered list of input/output cells which can contain code, text (using Github Flavored Markdown), mathematics, plots and rich media.”
I’ve got an ever growing tool set of F# data related functions that I’ve moved to a personal lib in my dotfiles and use in scripts, notebooks, etc.
Then you still get a digital lab notebook that ties together scripts, plots, and documentation, but the scripts remain usable standalone.
Edit: jinx!
(1) The calculational methods we used - could either be a set of mathematical equations or a description of the algorithms. (2) The results of evaluating these equations/algorithms for different parameter values. Usually some graphs, and some discussion of their meaning.
A Jupyter notebook is designed to replicate that process but make it easier because the figures are produced by code right there. Personally, all my notebooks include a discussion in the markdown cells what I am doing, and why. It includes discussions of the code. And directly from the code, some graphs or numbers, with a discussion attached.
With the script workflow, I would have two different files. One with the code, and one with the results pasted in. It's annoying when my primary goal is to develop and test the algorithms under discussions. Best thing is, if done right, my work is completely replicable. Just run the notebook again.
Just because some people misuse the tool doesn't mean the tool isn't useful.
This is especially useful with large datasets. Even if serialization is straightforward, if you have enough data (or the data is remotely hosted), loading it might take anywhere from 2s to multiple minutes, and even 2s is enough to get you out of the flow if you are working rapidly and want quick feedback.
It's also useful when one of your cells goes and queries a slow API for a bunch of data — I do this all the time with Datadog.
if your calculation is long running you would not be as productive as could be in notebooks
In contrast to Jupyter, you’re still working with plain text files and not JSON, and you don’t end up saving the cached data in the same file as the script.
In contrast to Quarto, Jupytext, etc., this is still just a code file and not a MarkDown file with code blocks. Not all editors have fully working “go to definition” etc. in MarkDown code blocks, and in any case, many people need a standalone script that can be placed in an HPC job queue after initial local testing is done.
For example, there are a lot of cases my team uses notebooks for proofs of concept where we make a large expensive call to load a large chunk of data, slice a small piece of it, iteratively try to reprocess the piece until you get the reprocessing to occur the desired way, validate it reprocessed correctly, and then extend the reprocessing the the rest of the data set. That can all be done after only making 1 expensive call. Further more, if the last cell evaluation fails, it just resets you back to the line before and you can retry it.
Can you do this with a script? Absolutely. You can write a script to download the data, and a script to process the data, and sub scripts for the individual steps. But that's not the path of least resistance; the path of least resistance involves you having to edit a piece and recompile everything and reset the entry point. Avoiding really makes it easier to brute force to the desired state ASAP.
Can't really do that in a script unless you're running TempleOS
If your script requires loading 12+ GB of ml models into a gpu before running anything at all this is the difference between a few seconds and a minute to see a change also if the output isn't text you can see the image or chart result inline to that code.