Jupyter Notebook 7
blog.jupyter.org
blog.jupyter.org
Much <3 to the Jupyter team. Github pages + Jekyll is performant!
It is rather the attitude or non-ideology of the Jupyter contributors/team, that causes things like posting on medium or telling people to post their questions in their discourse forum. It is also reflected in the licensing they chose for their ecosystem. Though usually they are friendly and helpful towards newcomers, it has to be said.
Still, for a large open-source project, there is no overhead in using a static site generator. And plenty of benefits.
The good news is that moving your stuff from Medium to such is easy. Up to you if you pick Jekyll, Gridsome, Gatsby, or something else. See (full disclaimer: my blog post) https://p.migdal.pl/blog/2022/12/new-blog-from-medium-to-gri....
Are they though? Does anyone actually use JupyterLab by choice?
From what I've seen people love Jupyter Notebook but find JupyterLab misses the mark (and this is certainly my experience).
Would you mind sharing the areas where you feel lab falls short and where notebook does it better?
I want to give notebooks a try.
I always thought people only still used vanilla notebooks because that's what people say they use, e.g. "I work with Jupyter notebooks" (even though that may well be in JupyterLab). So most regular users wouldn't necessarily know about JupyterLab.
What is your setup?
That being said, I'm glad they've switched course and continue to work on Notebook once it became clear some people preferred it to Lab. With some of the added features and the ability to switch between Lab and Notebook more easily, I may give Notebook another try.
That being said, I usually just stick to one notebook per thing.
(I'm sorry for the questions that could be answered from documentation, but I can't find the docs on this feature! I have been wanting to specify data cells in a notebook, like the markdown cells, and then reference their contents from a code cell)
I always use JupyterLab by choice.
However between JupyterLab and VS Code Jupyter, it’s VS Code every time. It’s just so much better.
There is literally no downside to using it over notebook, why would you prefer notebook at all?
the file browsers, the terminal, the plugins, ... so much better
I started strongly advocating for it pretty much immediately. The waste of space on the margins of the notebook view was (is?) awful.
If that is your only concern, it will probably be quite easy to write a user style sheet to fix it.
https://jupyterlab.readthedocs.io/en/stable/user/rtc.html
Here's a Dockerfile that enables it:
FROM jupyter/scipy-notebook:2023-07-25
RUN pip install jupyter-collaboration
ENV DOCKER_STACKS_JUPYTER_CMD="lab --collaborative"
docker build . -t jupyter-collaboration && docker run -p 10000:8888 jupyter-collaboration
For me, it used to be Jupyter Notebook. For reasons I cannot pinpoint, I never got convinced to JupyterLab. Sometimes I use Google Colab, primarily for sharing and using GPU for deep learning. Now, when I run it locally, I do it in Visual Studio Code (https://code.visualstudio.com/docs/datascience/jupyter-noteb...), since I don't need to jump back and forth between it and the rest of the code.
[0]:
- random unusable scrolling with vim mode
- gg scrolls to top of notebook rather than top of cell
- seemingly more-limited refactoring
- ipykernel headaches when I’ve already specified the project interpreter
- randomly cell contents get erased on cell execution
- wsl headaches (allowing firewall exceptions for each new project)
- windows jupyter headaches (having to manually terminate jupyter sometimes to quit the ide)
- sometimes the debugger gets stuck fetching variables before displaying them
- some kind of incompatibility between non-pycharm-created notebooks possibly related to nb format version so they can’t be read
- removal of (ui affordances for?) the cell mode for scripts?
https://github.com/emacs-jupyter/jupyter#org-mode-source-blo...
People told me to use extensions but none of them really actually work, including the installation process.
Jupyter has a habit of breaking extensions on version upgrades. jupyterlab 3 -> 4 is a good example of this. Maintainers have to modify their metadata and then run a script. While this is trivial, maintainers have to be aware of the version upgrade, find time to do the upgrade, test, and then deploy. It’s really frustrating being a version behind because of extensions you need.
Good thing they are using one side to put a debugger in (shown in the screenshot)
Note that I'm probably a freak, lots of my friends love their IDE's, but having having something that works for my particular brain and my eyeballs is a blessing.
Previously there was "Jupyter Notebook". Then they separately wrote JupyterLab (creating a brand new implementation of notebooks for it). Now, they're taken the JupyterLab notebook code and used it to replace "Jupyter Notebook".
I use it sometimes to turn a notebook into a presentation and it doesn’t work with Jupyter Labs
Is any real science done with this or is it the Powerpoint for PyCon talks?
Reactive notebooks are nice but I’ve accidentally reran slow SQL queries because I updated an upstream cell and that’s painful (using Pluto, ObservableHQ, and HexComputing).
In practice I never see or create notebooks that don’t run when you push the run-all button, it’s a well understood and easily avoidable issue. It’s probably a local optimum but I’m happy with them.
This led a lot of programming environments to where batch loading of the code is basically required. But "image based" workflows are a very old concept and work great with practice. Some older languages were based on this idea. (Smalltalk being the main one that pushed this way. Common Lisp also has good support for interacting with the running system.)
It is a shame, as many folks assume everything has to be "repl" driven, when that is only a part of what made image based workflows work.
The models run on a small cluster and/or a supercomputer, and the data reductions of these model runs are done in python code that dumps files of metrics (kind of a GBs -> MBs reduction process). The notebook is at the very tail end of the pipeline, allowing me to make ad hoc graphics to interpret the results.
Nowadays they can even be used for running models/analytics in prod with tools like Sagemaker (though I’m not advocating that they should).
Maybe you’re mistaking Jupyter for a different tool like quarto or nbconvert but your dismissive comment misses the mark by miles.
Interactive environment without compile nonsense is just too new for folks.
#+BEGIN_SRC python :results file
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
fn = 'my_fig.png'
plt.plot([1, 2, 3, 2.5, 2.8])
plt.savefig('my_fig.png', dpi=50)
return fn
#+END_SRC
#+RESULTS:
[[file:my_fig.png]] import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 2.5, 2.8])
Alright, saving the figure at 50 dpi first
plt.savefig('my_fig.png', dpi=50)
Trying a bit more DPI to see if that makes a difference
plt.savefig('my_fig2.png', dpi=150)
Oh, wrong numbers, forgot that the fourth datapoint was going to signify 100, going back to 50 dpi as well
plt.plot([1, 2, 3, 100, 2.3])
plt.savefig('my_fig4.png', dpi=50)
R.
No thank you.
Besides the ecosystem, what makes R better than Python or Julia for biostats?
The statistics built in are great. They're just there, less need to find a package (general stats, ttest, chi_squared test...). We tend to use the "tidyverse" packages [1] https://r4ds.hadley.nz/. Bio-python is amazing for manipulating biodata, but once the data is extracted and you need statistics, our scientist seem to use R. I really don't love R's syntax, but I get why they use it. I use python all the time for data wrangling (right now I'm pulling sequences from a fasta file to inject into a table).
Rstudio is like an IDE for your data. You can view the data tables, graph different things etc. If you try the first chapter of the R4data Science book, you can see how get up and graphing and analyzing quite quickly. https://r4ds.hadley.nz/data-visualize.html
Though at this point Python and R are necessary depending on what package/ algorithm you want to use.
There are some good packages for single cell analysis: We use "Seurat".
https://satijalab.org/seurat/articles/get_started_v5.html
Jupyter supports R now with an add in, so its less of an issue.
People use it for two reasons: a) because they need to get those graphs on the screen and this is the only way b) running ML code on a remote, beefier server.
The real reason is because it’s a much better workflow for data exploration and manipulation because you don’t always know exactly what code to write before you do it. So having the data in memory is really useful.
Or you just use jupyterlab and the problem is fixed.
As it is now, you typically wind up “programizing” your notebook once it does what it should so you can run in batch and so on.
Do you have a source of this, or is it something you dreamed up? Weird claim as none of those are my use case.
Jupyter notebook is neither the first nor the only implementation of such literate approach.
If some code is stable enough for reuse, you can make it composable as any other code: put it into the module/create CLI/web API/etc -- whatever is more appropriate in your case.
Advantages...no setup on the students side (+they get reliable compute remotely) and we can prepare notebooks highlighting certain concepts. Text cells are usefull for explaining stuff so they can work through some notebooks by themselves. Students can also easily share notebooks with us if they have any questions/issues.
I also use notebooks for data exploration, training initial test models etc. etc. Very useful. I'd say >50% of my ML related work gets done in notebooks.
I personally prefer when people share code as notebooks because you have code alongside the results. It’s really a good practice to use Jupyter.
Say you have a large file you want to read into memory. That’s step 1, it takes a long time to parse that big json file they sent you. Then you want to check something about that data, maybe sum one of the columns. That’s step 2. Then you realize you want to average another column. Step 3.
If you write a basic python script, you have to run step 1 and 2 sequentially, then once you realize you want step 3 as well, you need to run 1, 2 and 3 sequentially. It quickly becomes much more convenient to have the file in memory.
That is why jupyter lab is not the wrong name, it is a bit like a lab. Not meant for production use, but very good for exploring solutions.
Then I can easily put them into a Python file and import them from the notebook.
Easy peasy and very nice for iterative development.
https://income-inequality.info/
All the processing is documented with Jupyter notebooks, allowing anyone to spot mistakes, or replicate the visualizations with newer data in the future:
This is interesting! Looking forward to testing the purple theme I use [1]. I wonder if extension developers will need to maintain three sets of instructions now? JupyterLab, Jupyter Notebook >= 7, and Jupyter Notebook < 7
But even though VS code has poor support for notebook shortcuts (half the time a cell is in focus “the wrong way” so you can’t use ‘b’ to create a new cell) I find the convenience of just clicking a notebook file and see it rendered too much.
Regardless I am happy to see they have revived the project and am eager to check out the new release!
I want to build a template notebook, that has internal code to fetch data from a database, based on command line arguments and then run the notebook and then strip all the code parts and generate a beautiful PDF document.
https://saturncloud.io/blog/how-to-hide-code-in-jupyter-note...
I could be completely wrong here, just looking for insight.
Neither in Notebook nor in Lab can I click in the gutter to set breakpoints. The debugging panel is open, the documentation is clear (except that by default there are no line numbers and you have to activate that), but nothing happens.
Where exactly am I supposed to click?
That whole issue feels so stupid.
I quite enjoy jupyter lab otherwise, even if a lot of it is brittle and annoying.
(TLDR: some novice users in educational settings find the lab environment overwhelming.)