Hacker News Reader

> For myself (a professional data scientist) and every other...

The key qualifier is 'from a development perspective'. Notebooks do have their usefulness. I do some data-science related tasks but I'm mostly a developer, so admittedly that could skew my perspective.

> If you think of a notebook like a tool for literate programming...

I mean, that's kind of a selling point of notebooks, though...

> They don't integrate well with text editors

They don't integrate well with anything, not even other notebooks! From that alone I sense strong Word vibes.

> If you prefer using Spyder and RStudio with the console and plot window, that's great.

My usual setup is pretty much the same as yours with Visual Studio Code, but yeah, that was the point... So notebooks are at least somewhat competing in the same space as IDEs after all? Is there a subtlety I'm missing or "I prefer IDEs" and "Notebooks are not competing with IDEs" are in direct contradiction?

> keeping track of / taking notes on what I did, and displaying the results in a way that I and my colleagues can easily understand what I did.

Again, I'm not arguing with that, I was talking about notebooks as a development tool. I can very well see the appeal of notebooks as a presentation/storytelling/documentation format, and to an extent even I agree they have their usefulness there, and I certainly have no qualms with their usage in that niche. But I just couldn't bear to sit in front of Jupyter instead of VSC to code something, hence my strong wording.

Notebooks are primarily a discovery tool when you're exploring your data, deciding what ETL transformations and feature engineering to create and testing algorithms.

When you want to run stuff in production, you move to a regular code dev environment (or handoff to developers).

Now some things in business are infrequent enough that you can live with Jupyter for the once a quarter or so when you need to rerun updates but that's not the same as production code that runs daily/weekly.

My point is that notebooks are not a development tool, so it makes no sense to criticize them from that perspective. Nobody is asking or even suggesting that you develop software in a notebook, and I haven't met anyone who does it.

Spyder and RStudio are very much not oriented toward traditional software development and very much oriented toward interactive work between the text editor window and the console/REPL window. That is, they are fancy implementations of the Editor+REPL workflow. The fact that you can develop software in that kind of environment is not the point.

> Nobody is asking or even suggesting that you develop software in a notebook

It doesn't have a huge audience at present, but https://nbdev.fast.ai/ is doing just that. I'm personally of the opinion that those looking to use notebooks as a programming environment would be better suited by something like Smalltalk. That is, something that seamlessly blends between stored programs and the runtime. You can see some progression towards this in VS Code (I believe there's an issue about exposing variables evaluated in the ipython REPL in autocomplete), so I'm not terribly bullish on Jupyter-style notebooks being the next big thing.

> ...and I haven't met anyone who does it.

Really? Developing software in Jupyter must be almost as common as the "just code an Excel macro bro" approach.

Working on an analytics team myself, I strongly agree with everything in your reply. I just want to nitpick one bit:

>weakly recommend against writing production ETL pipelines in a notebook

I find prototyping and developing ETL code in notebooks to be the most efficient. Especially when dealing with new and unfamiliar data sources. The interactivity and feedback loop makes defining what an ETL process should be doing a lot easier.

That said, once it's working, everything should get moved out into a module with tests.

The problem with a notebook (that you don't have with a REPL) is the execution order is sort of arbitrary and changes don't propagate well.

A notebook is great if you don't change anything, but at that point it's basically a REPL with inline graphics, which is usually what I really want...

This is both a pain (state gets confusing) but this statefulness also a huge boon when rapid prototyping. You don't need to rerun calculations to tweak a single variable. Sometimes the data preprocessing can be really expensive.

It does take a certain discipline to make reproducible notebooks. I like to always end the day with a "restart and run all" if possible.

Right I guess what is needed is some way for cell dependencies to be known, so that all cells that depend on a changed cell will automatically update.

Just restart your kernel if you think something spooky related to state is occurring. This is the number one criticism I see of notebooks, but it is so easy to deal with if you are aware of it. The benefits of being able to rerun an isolated part of your code far outweigh the harms IMO.

Its done in Pluto.jl - Julia notebooks that have cell dependencies implemented (and rerun them upon change of variable’s value).

Maybe I am misunderstanding how the system works but I don't actually want that kind of Excel-like eager behavior like 90% of the time. Sure it's neat for simple computations but some cells take hours to run in my workflow.

Give me an indicator that a cell is stale but I don't want eager execution on all the time.

The problem with a notebook (that you don't have with a REPL) is the execution order is sort of arbitrary and changes don't propagate well.

How so? You can execute lines of a file out-of-order in a REPL just as easily as you can execute notebook cells out-of-order.

I like notebooks for eda and some prototyping but then move to vs code.

From my experience, the the problem is many people in DS positions or who start with notebooks are unaware of what editors are for and don’t have much grasp on code maintainability or collaboration.

They simply don’t know what they don’t know so their notebooks are spaghetti code mixed with tons of random things being displayed that was a sanity check for them at the time but now not even they know why they printed or displayed it.

Don't know about Jupyter, but been using Rmarkdown for as long as they are available. Recently, however, I am moving away from Rmarkdown and I don't miss it.

IMO their ideal use case is in tasks like communicating homework exercises to a teacher. Or for tiny stand-alone projects.

The problems they are designed to solve are, supposedly, numerous: reproducibility, literate programming, interactive exploratory analysis. But they end up getting in the way more than helping.

1) Reproducibility: nothing in the notebook is more reproducible compared with a simple source file. And you cannot really reuse anything within the notebook. Notebooks make re-running the written code more convenient for people not familiar with your stack. But, like mentioned above, it looses its appeal in any substantial project, where you will not be able to share the whole data that fuels the notebook.

2) Literate programming: it works only when you don't change anything, or make the notebook very abstract. If you do an analysis on a prepared dataset, add some comments, draw some pictures, then you have a "literate programming" document. But it lives only as long as nothing is changed. If you change the dataset it depended on you will likely have to change the comments in the notebook. You cannot just re-run it. And the comments added to the notebook - in my experience - nobody reads them, besides the original author. Almost every time somebody sent me a notebook I had additional questions about the results, which required me to interact and ask questions. It just adds additional burden on your part to describe what you do two times: once in the code, and another time in text. You can either guess what the recipients will be asking about and try to cover all corners, or simply answer those questions directly in a shorter amount of time.

3) Interactive exploratory analysis - notebooks slow it down. Writing simple functions in an editor and sending them to a REPL or simply exploring things in the REPL directly - both are faster alternatives. With notebooks you have to care about presenting the figures and pictures and comments from step one. In a typical exploratory analysis you are just trying things out without any idea of preserving them, before you stumble upon something useful.

4) On top of all that notebooks make things difficult to change. Say you have a notebook that loads 3 datasets, compares them with pictures and tables, transforms them, then compares them some more. And you decide to change the color of one picture. You either will have to re-run this notebook from the start just to change the picture. Or, alternatively, you start using cache, which, in my experience, is never a better option.

Just adding an alternative opinion.