What’s wrong with computational notebooks?
web.eecs.utk.edu
web.eecs.utk.edu
I don't get to use it in my current role, miss it a lot.
Sharing data? They had enough problems sharing their notebooks.
As someone who spends a substantial amount of time working with both modes (writing research code in Jupyter notebooks, and writing production code as python modules), notebooks scratch certain itches that IDEs typically don't even come close to. (Some recent progress on add-ons in Javascript-based editors is potentially interesting, because that might help marry the strengths of the two)
In my experience, in the evolution of code from Jupyter notebooks to repositories of production code as part of any project, there comes a "right time" to switch from the former to the latter. And this can typically only be learned with experience.
That is basically what I meant by knowing when to transition from one mode to the other.
Here's a concrete example (maybe somebody considers this an inspiring challenge?), to illustrate how notebooks are infuriating in their primitiveness, but still better compared to using an editor on source files: Imagine a beginner trying to write/learn a sorting algorithm, and who would like to keep experimenting with their code and observing what happens on examples, possibly profiling space/time complexity along the way.
To expand on my point above, there are actually three distinct computational use cases, not just two: Interactive learning -> Sharing insights with others -> Productionizing code.
Unknowable ad-hoc, unversioned spreadsheets running much of the capital of the company.
Notebooks tend to be the same way. It’s a simple GUI-ish was to do many complex analyses in a quick and dirty way.
And many of the arguments for not using Excel are the same as not using notebooks. Each is good at the initial data exploration stage, but are often abused and used in production when everyone knows it is a bad idea. But it still “works” so it is unlikely to be replaced.
(Especially when those that are working with the data don’t always have the skill set to build out a full production workflow.)
(For what it's worth, I feel the same way about people who try to send me RDS files with dataframes stored as R objects).
However, I think that whoever decided to name genes "OCT4" and "SEPT7" have to share some of the blame here too...
We just store the data tables in the project's database on a Postgres server. Then it's just a matter of pd.read_sql_query()
My background is programming (instead of data analysis & modeling) so I'm sympathetic to your idealistic "software engineering" view... but I'm also sympathetic to the academics' side as explained by Yihui Xie's blog post:
https://yihui.org/en/2018/09/notebook-war/
He's convinced me that criticizing non-programmers for using (or over-using) computational notebooks when it should be a "proper" programming language and deployment is like criticizing financial analysts over-using Excel to learn how to program VB or Python and re-write their spreadsheets into a "proper database" like Oracle or MySQL. That's just not reality. This divide between "end user tools" and "proper programmer tools" will always exist because there is no perfect tool in existence that serves the needs of both skill sets. Therefore, the programmers will always be able to say the data scientists or financial analysts are "doing it wrong".
I think this is very much off the mark. For sure plenty of scientists are poor programmers, but that isn't the reason they use notebooks. It is because:
They are not attempting to write something that will run everywhere, and often. They are either analyzing some data or doing rapid prototyping. For the latter, it's like criticizing someone who uses a REPL. It's just that the Notebook is much more powerful than a simple REPL that one can safely stick to it. Imagine you will do 40-50 prototypes and only one of those may end up worthy enough to make a product out of, and you don't know which one that will be. If you used a non-notebook environment, you'd give up in frustration by the time you hit the 15th one.
As you said: At the moment, there simply isn't an alternative that allows for rapid prototyping and is production ready. It's a hard problem to solve - there's a reason no one had solved it for decades (well before notebooks were a thing).
Had notebooks not been invented, you would have the same people handing you MATLAB code asking you to productize it.
Claiming they are beginners/novice programmers is off the mark. Peter Norvig started using notebooks for a reason, and no one would call him a novice. I do SW for a living, but when I need to analyze data and visualize it, I'll pick a notebook over "proper" SW tools any day.
I am very happy to be out of that business.
See Yestercode [1] and CodeDeviant [2], two tools that I specifically designed for LabVIEW programmers to refactor and test their code without expecting them to behave like traditional software engineers.
[1] http://web.eecs.utk.edu/~azh/pubs/Henley2016VLHCC_Yestercode...
[2] http://web.eecs.utk.edu/~azh/pubs/Henley2018VLHCC_CodeDevian...
This is a very minor question (and I am not concerned about risk to participants)--when you say they signed consent "in accordance with our institutional ethics board", are you talking about Microsoft, one of the two universities, or all?
It's not clear who the audience is. It sounds like most people who complain about them are software people and not researchers/scientists.
For someone like me, who once did computational research using MATLAB, and later analyzed data for my job, Jupyter is not worse, and is in most ways superior. Let's take your points one by one:
> Participants stated they often downloaded data outside of the notebook from various data sources since interfacing with them programmatically was too much hassle.
This was the norm with MATLAB, Excel and JMP as well, unless someone wrote code to autodownload (extremely rare - less than 1% of people did that). And if you are going to write code to get the data from somewhere, it's much nicer in Jupyter than in these other tools.
> Not only that, but notebooks often crash with large data sets (possibly due to the notebooks running in a web browser).
I honestly have not seen this, and the reason makes no sense. Your browser is not handling the data. The kernel is. I mean yes, if you try to load several GB of data in pandas, it's possible you will have problems if you run out of RAM, but this has nothing to do with notebooks.
> Once the data is loaded, it then has to be cleaned, which participants complained is a repetitive and time consuming task
This was as much a problem prior to notebooks as it is now. Notebooks did not make this any worse.
> Explore and analyze. Modeling and visualizing data are common tasks but can become frustrating. For example, we observed one participant tweak the parameters of a plot more than 20 times in less than 5 minutes.
It was even worse with MATLAB. Ditto for Excel. JMP is a bit nicer for visualization, though.
> Notebooks do not have all of the features of an IDE, like integrated documentation or sophisticated autocomplete, so participants often switch back and forth between an IDE (e.g., VS Code) and their notebook.
It may be better now, but this was a problem in MATLAB as well.
> While it is easy to share the notebook file, it is often not easy to share the data.
This is as true with MATLAB, JMP, etc. A lot of the complaints about it being hard to reuse notebooks is because notebooks at least attempt to be reproducible, and thus many more people attempt it. Prior to notebooks, I know almost no one who tried to share MATLAB analyses, because it was such a pain to do so.
> Notebooks as products. If a large data set is used, as one might expect in production, then the notebook will lose the interactivity while it is executing. Also, notebooks encourage "quick and dirty" code that may require rewriting before it is production quality.
I suppose some people are trying to make products out of notebooks, and this is where all the recent grief I see is coming from. I do not think it was the primary goal of notebooks, though. They were meant for data analyses and prototyping, not for production use.
Kudos for looking at real people's work and surveying it.
I wonder how much workflow could be improved if researchers would be temporarily paired with developers - who are generally better at modularising and removing friction in their work.
Personally I believe that a bit of clean-code discipline and following known best practices could solve couple of those pain points.
It's also true some could be improved by rethinking how notebooks work; ie. being able to specify input/output of notebook so it can be used as a library; detaching runtime data from the code so it plays better with version control/publishing; maybe even more radical ideas like adding visual/flow view that helps with linking elements; adding built-in excel-like sheets that can be queried/manipulated could also be interesting; built-in, first class support for relational database (sqlite) could also be a big win.
There are many interesting developments happening in this space and there seem to be some unexplored ideas waiting to be tested out.
I found your defense, which basically just says "It wasn't better before - so, no critique allowed?", considerably less valuable than the article.
This is true nowadays with Jupyter, because it is smart about truncating output. But it used to be possible to OOM the browser by e.g. printing in an long-running loop or displaying too long of a list/table.
But I guess if you're manually printing in a for-loop, then I suppose you could make the browser crash - whereas tools like MATLAB wouldn't.
People who design and implement features in notebooks. The conclusions in the blog post and research paper are clear that improving these identified problems could improve user experience.
What are your thoughts on the best way to address these things?
I'd also love code completion in notebooks.
I think the cleaning and code reuse problems can easily be mitigated by putting functions into libraries and using auto reload.
My normal workflow is hack something in a notebook until it runs, then refactor and put in a library I import with auto reload. I work on production ML and I use this for both software development and research.
We use it exclusively for our Data Science and find that it ameliorates all of the pain points you highlight in your article.
Have you looked into the domain of "research data management"? Concerns such as "archival", "security" or "share & collaborate" are core to this research domain:
In academics, there's a trend to prepare a "data management plan" up front that creates awareness about these concerns. They are even a requirement in order to get funding:
i.e. https://dmponline.dcc.ac.uk/
So, it's a bit odd to see a study that's focussed on a single technical tool yield the same concerns... but not making that jump to a larger, existing framework on information management.
Looking at the authors, it seems you are located at Oregon State University. A quick DuckDuckGo search yields this service from your colleagues at the University Library:
https://guides.library.oregonstate.edu/dmp
With the context of notebooks themselves, I think the study reflects on "if you have hammer, every problem looks like a nail." Notebooks aren't the only powerful tool to work with data. I think many of the same concerns could be raised with Google Sheets or Excel with heavy VBA scripting. Like others said, this is not a new problem.
Notebooks do have a place in the bigger process of doing iterative research based on data mining techniques. They can help to formulate more accurate questions and perform quick tests without the friction of having to set up complex environments. Moving on from initial data exploration, it's up to the researcher to use a formal method and tools that do mitigate those concerns. RDM is all about providing tools and mitigating (legal) liabilities as far as "what do you do with your data?" is concerned.
1. Checkpoint the interpreter state after every cell execution.
2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there.
I can't tell you how many times I've seen accidental persistence of dead state waste hours of people's time.
At the same time, I do like the idea of an append only notebook where you can:
1. Only run cells in sequential order
2. Only edit cells that are below the most recently ran cell.
Thankfully you can enforce it through code practice and the notebook is relatively guaranteed to be "run all"-able. You will need to refactor it after the initial dirty run, but at least it's easy to reason about.
Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations?
I don't think so. Vector operations require the data to be aligned to whatever the vector size is, no? E.G. 16-byte vector ops require the data to be aligned to 16-byte, etc... At least that's my understanding.
I suspect that a big part of my annoyance about the current state of the data space is that parts of the ecosystem were designed with the needs of data scientists in mind, and other parts of the ecosystem were designed with the needs of data engineers in mind, and it's all been jammed together in a way that makes sure nobody can ever be happy.
In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing?
Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.
Colab has a nice feature that's close to this: Insert -> Scratch code cell
there are quite a few very useful ones, my favourite being collapsible headings: https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...
With sections of the history easily replaced.
> So you'd want to run `del foo` on the old name.
And then delete this line, because if you leave it in it'll break when you try and run the file all the way through.
> Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.
Do you mean by treating them as append only and never rerunning any cells?
> In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing?
The notebook has no idea if it can or can't, but that doesn't mean that leaving it in is good it's simply the only realistic option.
Also polynote by Netflix, as a user below mentioned.
(don't reply, and I'll delete this)
* * *
More generally, Observable solves or at least ameliorates every item in the linked article’s list of complaints. (In 2020, any survey about modern notebook environments really should be discussing it.)
I found the article quite superficial. More like “water cooler gripes from notebook users we polled” than fundamental problems with or opportunities for notebooks as a cognitive tool. I think you could have learned more or less the same thing from going to whatever online forum Jupyter users write their complaints at and skimming the discussion for a couple weeks.
I guess this might be the best we can hope for from the results of a questionnaire like this. But it seems crazy having an article about notebook UI which makes no mention of spreadsheets, literate programming, Mathematica, REPLs, Bret Victor’s work, etc.
From the title I was hoping for something more thoughtful and insightful.
In practice I think taking care not to accidentally shadow variables is much more important: this dependency business only makes sense once you have a clear idea of what you need and by that point you are mostly done anyway.
[1] https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...
In practice, I find it to be dramatically better than previous notebook environments for data analysis, exploratory programming / computational research, prototyping, data visualization, and writing/reading interactive documents (blog posts, software library documentation, expository papers ...). It has a lower barrier to starting new projects, a lower-friction flow throughout
I find it better at every stage of my thinking process from blank page up through final code/document, and would recommend it vs. Jupyter or Matlab or Mathematica in every case unless some specific software library is needed which is unavailable in Javascript. The only other tool I really need is pen and paper, though I also use http://desmos.com/calculator and Photoshop a fair bit.
We were gonna buckle down, pay the cards down hard for a while, 'color' our money so we both had discretionary spending separate from, say, the power bill. She had much more Excel experience than I did so she worked up a spreadsheet.
It was bad. We had worked up some 'fair' notion of proportionality and she basically had no spending money and mine was pretty bleak. So I redid the numbers from scratch with split that was better for her. In the new spreadsheet she has much more spending money and... hold on, I've got a bit more too? I looked at her spreadsheet repeatedly and I never did figure out where a couple hundred bucks got lost. I went back to sanity checking mine instead to make sure I wasn't wrong. It checked out.
I wonder sometimes how often small companies discover they've been running in the red instead of the black, because some cell got zeroed out, a sum didn't cover an entire column, or embezzlement is encoded straight into the spreadsheet without anyone noticing.
There's gotta be a better way.
This is a really interesting insight (actually obvious when you think about it). I'm currently working on a spreadsheet app and these kinds of observations are very interesting to me. I guess things like named cells/variables will help (instead of using $A$4 etc.). Range selection could also be more intelligent (it could actively warn you if a range selection seems to be missing a few cells of the same data type). Do you have any other insights here?
A good notebook (again, similar to early literate programming tools) should help you piece the final execution flow back into the procedural flow needed for the given compiler/interpreter, but it probably should still let you rearrange it to best fit your narrative/thesis/arc.
> RunKit allows you to rewind your work to a previous point, even filesystem changes are rewound!
Hot reloads can become very expensive. Especially when it comes to computationally heavy tasks that notebooks are built for.
If you decide you want hot reloads by default, it'd mean each time you click on a cell and then click on another you'd be restarting the whole notebook.
If you had massive datasets you were loading or other args that you were parsing manually or at prompt, you'd have to go back and do all that. Don't even get me started on the operations you'd have done with those dataframes prior.
I think it is a good thing that notebooks separate instructions and re-execute manually by default. The cost of the alternative is just too high
Maybe add a "lock" toggle so a user can block a cell from being automatically executed? The heavy numeric setup tasks could then be gathered in a few cells and locked, leaving the lighter plotting & summary stats cells free to update reactively.
Toggle a whole environment and intrepeter's behavior? Do you know how much architecture that would involve? That's like trying to tell IDLE to be able to both delete or keep your variables on exit, or the JVM to have a toggle switch for memory and garbage management.
Why doesn't the developer make themselves useful and simply write a save function that freezes their buffer variable values to a text, json or SQLite file that they can read from or stream rather than trying to set back a whole community years of progress in an effort to accommodate perhaps entitled or lazy devs.
Can you even imagine the architectural costs of trying to accommodate streaming data and timestamped data as opposed to you just writing your own stuff to file?
Unless it takes more than a few seconds to run a notebook, rerun every cell up to the point you're editing, always.
And then if it does take minutes, and you find yourself in an unexplainable rut, then run the entire notebook, and get a cup of coffee.
We (at Nextjournal) tried doing the same for other languages (Python, Julia, R) and felt that it didn’t work nearly as well. You often want to change a cell at the top e.g. to add a new import and it can be quite annoying when long-running dependent cells re-execute automatically. I now think that automatic execution of dependent cells works great when your use case is fast executing cells (see observablehq) but we need to figure out something else for longer running cells. One idea that I haven’t tried yet is only run cells automatically that have executed within a given threshold.
I hear a lot of complaints about hidden state but I think it’s less of a problem in reality. It’s just a lot faster than always rerunning things from a clean slate. Clojure's live programming model [1] works incredibly well by giving the user full control over what should be evaluated. But Clojure's focus on immutability also makes this work really well. I rarely run into issues where I'm still depending on a var that's been removed and then there's still the reloaded workflow [2].
Overall I think notebooks are currently a great improvement for people that would otherwise create plain scripts – working on it is a lot quicker when you have an easy way to just execute parts of it. Plus there's the obvious benefit of interleaving prose and results. That doesn't mean we should not be thinking about addressing the hidden state problem but I think notebooks do add a lot of value nevertheless.
[1] https://clojure.org/guides/repl/introduction
[2] http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-r...
https://github.com/topshell-language/topshell
* Information only flows downwards.
* Computations are cached.
* Things are automatically recomputed when the values they depend on change.
* Things with effects is instead cleared and awaits confirmation before running.At the time, my workflow was to use two different notebooks for everything: foo.nb and foo-scratch.nb. I’d get things working a piece at a time in foo-scratch.nb, not caring at all how it looked, not having to worry about leaving extra output or dead ends of explorations lying around; then the refined cells would be copied over to foo.nb, which would get pristine presentation, and which I could run top-to-bottom.
This workflow worked pretty well for me: very clean reproducible output, with the ability to easily refer back to all the steps of how I’d derived something, along with copious detailed private notes.
I never had to use it but I’m pretty sure each cell even had its modification time stored in the metadata in case I wanted to view a chronological history.
slides: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...
doesn’t solve the state management testing or tooling issues though, but commits are slightly less awful.
As the article (and the many others like it that have frequently cropped up as soon as IPython Notebooks first started ramping up in popularity) points out though, a lot of newer users don't have the discipline to ensure that they're not jumping around too much. It's not a problem for them in the immediate term since they know how the state ought to work, but then it becomes a mess when they try to share it with someone else (or to run it themselves again 3 months later).
The challenge though is that the data analysis workflows that it allows are unbeatable by any other tools I've tried. In the end, it may just be that it's the worst form of data programming except for all of the others that have been tried.
Biggest tip I have is to turn auto reload on, then write the bulk of your code as modular functions and call functions within your notebooks. Keeps the notebook tidy and it’s easier to push your code this way.
It’s also easier for sharing since most people viewing your notebooks (mentors, people outside your team) are interested in results/artifacts such as metrics, generated text, images, audio, which notebooks display well (not your code).
The main question is: what are the alternatives, for data exploration (and sharing its results). Similarly, for data science tool demos, Notebooks shine.
IMHO the problem is not in the notebooks, but in how they are being used (i.e. the workflow). By writing scripts in py files, and using notebooks only to show their results (processed data, charts, etc) we get the best of both worlds.
The only build-in problem with Jupyter Notebooks is JSON, mixing input and output (and making it pain to work with version control). But here RMarkdown (and a few other alternatives) work well.
I'm currently at uni enrolled in an AI/ML degree, and there are a lot of people with no previous exposure to programming. It's just that most people don't know that these things are possible, don't want to learn another tool (IDE) and are not interested in longevity of the code, just in the results. This shouldn't sound like me complaining, I totally understand. I think a lot of the stuff could be solved with just better tooling, but a familiarity with software development is definitely helpful.
Also a while back streamlit (https://www.streamlit.io/) was here on HN and since then I've been meaning to try it. I think this could be a good approach to bring together the best of both worlds.
This approach is arguably more effective than wasting time trying to refactor everything into a library of functions.
Programming-as-crafting needs to be more of a thing. Not everything is written to be long-lasting. Even HN ushered the ugly code into hook functions that weren't shipped with the main codebase.
It eases most of the pain regarding version control. You can use it as a 'git filter', so only inputs would be shown in diffs and committed (and also works with interactive adding!), while keeping outputs in your working tree.
Absolutely. JSON notebook format makes it very hard to do code reviews, merge in remote changes etc. After being frustrated with lack of solutions, I built ReviewNB[1] specifically to do notebook code reviews on GitHub. Alternatively, Nbdime[2] is also a nice open source library to see diffs locally & merge in changes.
After the exploration and preprocessing stage I personally don't see much benefit of the notebook model, training/evaluation and any meaningful visualization takes forever anyway, that means I need to cache and persist intermittent results. With that it doesn't really matter all too much if I work on it in vim&pdb, an IDE, or Jupyter.
In a good data-oriented IDE like RStudio you get to do all of those things and write code which can be saved as plain text and can be version controlled well under git which you can't do well with Jupyter.
R folks have to be the best indicator in this case because they have access to a good IDE and they have good support for Jupyter. Their use is overwhelmingly in plain text files in RStudio, a small portion of rmarkdown notebooks and pretty much no one user R in Jupyter.
Notebooks give me some of the interactivity but the experience degrades significantly.
The spyder IDE seem like an okayish replacement but some of the library I use expect you to have html display (within a notebook) to give you full functionalities which is not yet available in spyder.
Rstudio gives you the experience of a classical IDE + easy data exploration which I found to be productive from the exploratory stages (where I need to see my data and the effect of my code) to the clean-up phase (where I refactor my file).
Same with the results - once you've coded something (perhaps outside of a notebook environment) and obtained results, verifying that they are what you expect is much more efficient to do in a notebook.
If I'm writing a library or adding new features to one, or writing tests I'll use PyCharm sure thing, otherwise the notebook is a quicker way to sketch prototypes and always have a kernel with preloaded datasets and pre-imported stuff ready at hand. I don't want to wait 10 minutes to just load the data every time I want to check if my new function works well on it at big scale. That's one of the most important bits.
PyCharm is a clear winner at actually writing code that you won't throw in the bin 10 min later, and once you know what to write.
Debugging? Don't remember ever using PyCharm despite the fact it exists... either pudb or python-devtools or something else. I'd just write tests and things start working in the process. And btw you have pdb debugger (some weak version of it) in jupyter if you really need it. Docstrings? Press tab twice in the notebook. Or keep PyCharm open on the side so you can cmd-b. Profiling? Never a pycharm builtin, maybe something like flamegraph but an external tool anyway.
The Python IDEs for data science are mostly garbage - if you have any recommendations, I'm all ears because I really don't like notebooks but still keep switching between jupyter and vscode depending on what I'm working on.
Can it show me inline plots and allow me to embed rendered formulae written in LaTex, images and video in between lines?
This is the reason people like notebooks.
I love notebooks, I work fast, line by line I execute commands and I immediately see the output (dataframes or graphs). For complex code I have an editor open (in jupyter-lab or vscode) for some functions and classes. But the main developing is done in the notebook, anything that ends in a module start in my notebooks.
As a biologist that learned to program after 30 I just don't understand how you can develop data processing code without such a close handle on dataframes and without checking in graphs/visualizations if your code does what you expect. I don't see how I would do that in pure vscode of other IDEs.
I also don't understand this sentence: "Once the data is loaded, it then has to be cleaned, which participants complained is a repetitive and time consuming task that involves copying and pasting code from their personal "library" of commonly used functions." What is the alternative? Not cleaning the code? And why copy and paste when you can perfectly fine have your own shareable module on the side? I guess most notebook users do some kind of hybrid development.
Here’s a post I wrote just a few weeks ago describing some of the conventions that I established for myself over the course of 5 years:
https://jessimekirk.com/blog/notebook_rules/
I suspect that a lot of the conventions I describe help mitigate problems described here, some of which should be strictly or optionally enforced by the notebook instead of the user.
(The site’s very much a work in progress, so expect to see odd and broken things if you go poking around.)
I just started working with the Guix kernel for more easily reproducible and reusable notebooks. I suppose that's an alternative to using a conda environment.
See here for an example: https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a9...
Bostock had lots of accessible D3 examples before, now it's all on that "platform".. sure it's slick, but overall loss for the ecosystem IMO.
Editing an ordinary Python file, if you insert the comment "# %%", you turn everything between that comment and the next "# %%" (or the end of the file) into a code cell that can be submitted to the ipython kernel, just as in a Jupyter notebook. The editor splits into two halves, the left half your Python file and the right half the Jupyter notebook window with submitted code and formatted output (e.g., DataFrames look pretty, plots display normally, etc.). When you're done running everything, you can export the result as a Jupyter notebook. Because you're editing an ordinary Python file, standard features like version control and importing the file you're editing into other files (you cannot normally import .ipynb files IIRC) work normally.
And of course since VS Code is a real editor/IDE, you can double click a file and have it open right up (no resorting to a Terminal to start your Jupyter session) and you get syntax themes, a built in Terminal, a git UI, code snippets, documentation on hover, vim mode if that's your thing, etc.
The only downside I've found is that the Python extension doesn't incorporate ipython's autocomplete in its own autocompletion, but that's a small price to pay for getting to treat .py files as notebooks.
In my mind, notebooks should be married to a functional style of programming, where you use the notebook's markup to thoroughly explain and document your functions. Below your "function definition" section, you keep a "trying things out" section where you actually plug the data into your functions for debugging/visualizations. You can't shoot yourself in the foot with variables because all the work is done in your function's lexical scope. You can shoot yourself in the foot with stale function definitions, but a good notebook interface gives you the ability to clear function definitions and run groups of cells, so you can make sure you always run your functions in a group that starts with a "clear function definitions" cell.
When you are done, you just cut the "trying things out" section into a second notebook which references the functions in the first and viola, you've got a very well documented library of functions, and a new work notebook where you can freely polish your visualizations/whatever.
One category left out of the list: testing of data pipelines (c.f. great expectations).
The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't show the output without showing the TeX code. Importing the TeX interpreter is quite non-standard and means that my notebook doesn't play well with the public servers. I also got burned by some kind of permissions issue, so that all my charts ended up being invisible to read-only users.
The second crack -- I can only look at the code from within my own jupyter server. The source is buried in a very noisy json format.
The third crack -- Who wants to write code in the impoverished browser based editor provided? How many times have I deleted a closing brace that was automatically inserted incorrectly? How can I do a global search and replace?
The fourth crack -- I can't test my code unless I include all the tests in the notebook!
I'm complaining. I realize that I don't have anything constructive to offer, and I'm really a beginner. However, I think some of my disappointment is justified, as I think it was reasonable to assume that I could build my notebooks to be next level presentations.
In the end we dumped it all and turned the good parts into a sane CLI tool which ingests data and dumps out Bokeh plots. At some point we'll throw a Jenkins front end on it, but the current approach seems to be working fine.
The benefit here is that now your code and be used inside the jupyter notebook, and also inside a webserver say.
1. kdb is "too obtuse" for many and python glue makes it more amenable
2. I can still have kdb functions in source code and call them from jupyter with pyq
3. I can do most of my "editing" in emacs to the kdb back-end, write python "libs" for parsing results, and just use jupyter as a fairly thin presentation layer
4. I can share notebooks with analysts who run the same jupyter server virtual env, so finally we can share notes
Will this add more value than just using kdb? Time will tell, hard to know right now.
I agree that the "ide" experience absolutely sucks.
For part of 3, there is global search and replace accessible either via Edit -> Find and Replace or Esc-F, and it includes case sensitivity toggles, regex search, and the option to change the current cell or globally.
For 1, I think there are some plugins that can make that tidier, but I've generally just accepted that if the people I'm presenting to don't want to see code then I'll just need to make a set of slides out of the whole thing.
You should be able to see a blue vertical bar to the left of every code cell. Click that bar to collapse the cell. You can do this for both the code and the output. I know this works for Jupyter Lab, but I don't know about legacy notebooks.
I get the impression that people are creeping in the direction of trying to do everything with one tool, which sounds like it would end up in the same swamp that Eclipse went into. Sometimes, you need to use different tools for different tasks, and not everything should integrate. Just my opinion.
1. POC/MVP: Showing that what you want to do will work before making a full structure. 2. Creating PDF/HTML documents with code and output. 3. Exploratory data analysis and visualization.
I think many of the data scientists in the article go well beyond what a notebook is. A notebook is where you start, but should never be a production tool.
1. They are stored by default in stupid json files instead of plain source code with comments.
2. The text editing interface inside the browser is horrific and very difficult to normalize (e.g., disable "smart" closing of parentheses, disable the capture of classic unix copy-pasting, etc).
A) VS Code / IDE needs to be the primary editor B) Results are not stored with source C) Export (build) allows packaging for whatever platform.
Python notebooks especially also use some crazy mutable APIs. In general notebooks align with other code written by people who aren’t usually software engineers building production systems. They’re much more about getting things done, APIs and tools are less questioned, a lot of pain is swallowed because PhDs have plenty of time to write a few lines of code. I don’t want to sound disparaging towards these people, it’s just a different set of tradeoffs from writing production grade software.
I first saw jupyter notebooks when my sister (physicist, non-programmer) used it for analyzing economical data with pandas. Run-time for the full data set was half a day (and IMHO for that analysis SQL would have been better suited). I understand that as a non-programmer it looks alluring, but once the language proficiency is build up, why not use an IDE and run the code on a shell?
A common example is that a huge text file containing experimental data gets parsed in the beginning. Then you have to explore the data step-by-step using all kinds of visualization and analysis such as Fourier transforms, curve fitting, etc.
If you simply put everything into one giant python script, for every step you have to re-run the entire thing which takes forever. Of course you can speed things up by writing intermediate results to disk, but this adds tons of boilerplate code and is quite error prone.
One alternative would be to write individual scripts for each step and read them into an interactive REPL shell. However, then you still have to somehow record the proper execution order if you ever want to repeat the analysis.
If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes.
Storing/loading to disk is not a good option because all the data that needs to be stored is not yet determined until the exploration is finished; If I write code to save/load A, then I need to change (and test) it after I'm done with B and now want to experiment with C, and it all becomes even more complicated when I need to add an extra step and data field to step A and rerun everything. Deciding what data should be stored in what format is something that you can do in 'productionizing' the code after you've done the exploratory analysis.
REPL is not a good option because it's not convenient to save and replicate the code that got you to the current REPL state.
The other aspect is that visually 'debugging' intermediary data through various plots is not conveniently possible in IDEs. I could generate some picture files in a folder or possibly an HTML 'dashboard' to see the results of my most recent run but that takes extra code and effort, and the results aren't immediately in my face like in a notebook.
With what you describe, intuitively I would use a library that allows me to store&load data per step (with verifying the structure matches), or pass it in-memory. Think JSON (yeah, slow) or something like protobuffers. That way I could do both
> store(A(read(input)) -> file); store(B(load(file)) -> file2)
during development (or in case B is in another language as A), and in production just
> B(A(read(some_other_input)))
But yeah, that's just my intuition of course. Maybe I'd be a bad data scientist.
However, can't you just experiment with smaller data sets? That's what I usually do if processing is slow (e.g. instead of parsing 10GB of log files, I'll just do 50MB to verify the processing pipeline works, and once that's it, run it on the full 10GB and grab a coffee while it runs). Not an option for data science?
The intended usecase of these notebooks is in scenarios where the main output is not the code and not a particular set transformed data, but knowledge gained during a 'computational exploration' of that data. With that knowledge in hand, you can then build 'productionized' code with different methodologies (possibly but not necessarily using or adapting large parts of the code in your notebook), if that's needed - and in such data analysis scenarios it often happens that it's not ever needed.
Sampling a subset of the data sometimes works. Sometimes it would alter the results substantionally and drive the exploration in a wrong direction; questioning and verifying assumptions is important, and it can be a big difference if all A's are also B or only 99% of them are.
I can't just upload random data to some cloud service to work with it, also I can't upload data if it's too big. Often the data that's valuable is very sensitive.
0. They try to be "be-all, end-all" proprietary container documents, so they lack generality, compatibility and embeddability. It would be better if live code try-out snippets were self-contained and embeddable in other documents: HTML, other software, maybe PDF, LaTex or literate programming formats. Maybe there should be standard, versioned interpreters for each kind of programming language in WebAssembly and cached for offline usage by the browser for inclusion in documentation, papers, etc.?
1. For prototyping, it is better to have try-out live code (and/or REPLs with undo) for prototyping like what is Xcode/iOS Playgrounds for Swift or ReInteract was for Python.
2. Computational notebook software, that I've seen, are terrible, complex, fragile and messy to install. The ones I've seen make TeXLive look effortless by comparison.
3. Beyond replicability what goal(s) are CN really trying to solve?
3.0. For replicability itself, why not have a GitLab/BitBucket/GitHub repo for code and a Docker/Vagrant container one-liner that grabs the latest source when built? Without a clear, consistent and simple build process, there is no replicability, only wasted time, headaches and fragile/messy results.
3.1. Are CNs "hammers" for "nails" that don't exist?
This would be incredible. Even better, the output from the code (like graphs) should be able to be embedded in the paper. You have no idea how many papers have errors in the code that generated the graphs/statistics/etc. and nobody can tell because the authors rarely release the data, let alone the source
Then for HTML assets (and CSS ones too), perhaps a hint on asset-linking tags (a, script, link, img, audio, video, etc.) there ought to an offline-priority attribute to help the browser decide what to throw away when clearing cache the regular way or evicting items from the cache, while being able to leave some things deemed vital when not nuking the entire cache. Yes, websites could be goofy and game caching mechanisms, marking everything "vital" like for 0-pixel image cookies but I'm sure someone would make an "RBL" (real-time blackhole list) system of which priorities on which websites to ignore.
Related aside: There's a lot of common frameworks, libraries and bits that could be cached user-side, with the trick either to a) herding web devs to de-fragment their CDNs, which could create SPoF's or b) changing the standard allowing multiple SRCs or HREFs for high-availability/less bitrot to preserve both choice and encourage de-duplication of common assets. [0]
0. https://html.spec.whatwg.org/multipage/links.html#attr-hyper...
Their observations bring to mind the benefits of watching people program on YouTube or video where you learn a style of working you may not even have considered.
However there is one other issue that is not on the list: because a notebook is meant to be read or shared, I always feel like my work is public and feel less inclined to play around and just take a look at things. When I do “transfer” my work to a notebook, it’s only surprising or interesting things that suppress the discovery process.
This way I'm sure that "library code" that I'm editing in parallel in a real text editor is up to date in the notebook and also solves the limits the confusion due to run-out-of-oder problems.
The overall workflow is something like this:
1. explore using thing.<TAB>, thing?, and %psource thing
2. edit draft code chunk or function
3. when chunk 80% done; move it to a module
and replace it with an import statement
4. press 00 to re-run everything, then GOTO step 1
Sometimes I keep around the notebook as a record for failed explorations or as a "test harness" for the code, but most of the time it's throwoutable since all the useful bits have moved into a normal python module/script under version control.
import mylib; importlib.reload(mylib); from mylib import foo
In order to do notebooks properly — you need:
1. discovery (Ideally static discovery) of all the state the notebook needs, and the bulk of state the notebook will/could manipulate during its execution. Your container needs to intercept the filesystem and the networking apis that will be invoked so that a determination of the state that results from these operations can be observed by the runtime and shimmed appropriately for reproducibility and for performance optimization
2. The notebook (and the runtime inferred model of all the required inputs) needs to be repo stable — I Should be able to write a notebook app that reads from the file system on my development host, deploy it somewhere, and the runtime should take care that wherever that however that post deployment file system read is implemented matches my local development semantics
3. Pplatform level dependency graph needs to exist to model re-execution requirements automatically — incorporating code changes and external state
Apple could build this And “notebook-os” would be the correct conceptual framework for it ... anything less is always going to leave us severely wanting
The Python world gives a good example about the sheer complexity of a notebook infrastructure. The is IPython, there is Jupyter Notebook, JupyterLab. There is even stuff like the SageMathCloud (nowadays called CoCalc) which is basically a web GUI to a VPS combining command lines and various notebooks. And hell, most of these web based interfaces try to make sharing easy.
Mabye we should start comparing these (mostly OSS) tools to the traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better, but they lack all the tooling we get for free in the web.
I suspect 90%+ of Python Notebook work is done in Jupyter/Jupyter Lab (or things built on it like Google Collab/Kaggle Kernels).
traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better
I've done 10s of terabyte analysis on Jupyter (Spark backend) and I personally know people doing petabyte work on it so this seems doubtful.
It's easy to screw up a terminal application in data science when dumping a large array. Many REPLs cannot handle this properly (and CTRL+C won't work). It's easy to test this: What does your favourite notebook do when you call some command such as (pseudocode/python here)
print(range(int(1e7))) # or 1e8
It already eases a lot of pain you may have in comparison when setting up jupyter notebook without the knowledge of a software developer.
[0]: https://nteract.io/
I definitely agree with most of the other points though.
For me, notebooks are a super handy way of visualising and sharing results during meetings, and it's difficult to imagine a more convenient alternative.
This code + explanation would not work I guess.
With a couple of UML diagrams, still the best option.
Interesting study, I like the mixed-method approach. A quick glance at the industry of the participants suggests that there might be a bias towards structural data (which I think is actually acceptable as that makes up a huge chunk of the non-academic ML-Notebook work) Edit: The authors acknowledge this in the "Limitations".
The cells can then be individually executed in an ipython shell, or the entire script can be run with the regular python interpreter. This makes it easy to tweak the individual parts without having to re-run everything. In contrast to jupyter notebooks you still end up with a valid python script that can be easily version controlled.
I just wish that I could use vim instead of the Spyder IDE.
It lets you do linear execution of blocks like in Jupyter, but in a normal .py file. Obviously more lightweight than Jupyter and you get to use your regular editor.
We are still early access, but if you are interested in an invite just let me know. My email is filip at deepnote dot com.
If this information isn't private, what sort of business model do you use? I take it you'll have a SaaS subscription model? I see it's free to use now, but how does your company plan to make money (especially taking into account the cost of the cloud hosting Deepnote requires)?
Our goal right now is to build the most amazing data science notebook. We need a lot of feedback to get there, that's why we are keeping it free. But since the servers also cost us something, we haven't opened up Deepnote to the public just yet.
Once in GA, we know we can support students on a free tier almost indefinitely (it doesn't really cost that much) while offering more advanced features on a subscription model for teams and enterprises.
General: https://news.ycombinator.com/item?id=18336202
Version Control: https://news.ycombinator.com/item?id=21661013
This is intended to be a Zen-like Koan, so take it as you will.
Disclaimer: I wrote large part of treon.
Streamlit.io
By the way, I'd be interested in your thoughts on https://news.ycombinator.com/item?id=22083468.
The awful control flow syntax makes reading longer scripts pretty much impossible. Plotting is very clunky and by default produces output files that are essentially unreadable.
Of course one can somehow work around these issues, but it is much easier (and free) to just use python.
You need to pick the best tool for the job, and often times in machine learning that tool is a notebook.
The world is so much better with you alive. So is the founded tool of computational notebook. Not sure I read it covered R notebook which is really good to share info and analyst. Just wonder how to use it better.
Of course they can always improve on it. But I would promote more expansion - How about a lisp notebook, a clojure notebook, a js notebook and a forth notebook.
The real problem is can you have oo notebook ... it is more “serial” and graphic and data. But not for the “messy” class or trigger Based system. Hence if I may, the real problem is the scoping. It is so hard to visualise a live oo system. Unlike a live functional or even a stack based system.
It is not life that is the problem. Even useless life has its use, as long as it is alive. But if it is not reaching there an alternative may have to think about. Just like we cannot be there we send in our voyagers outside solar system.
Be long and prosper.