Introduction to Pluto.jl
juliafordatascience.com
juliafordatascience.com
Also, big "ugh" to browser-based tooling. I want to browse webpages in my browser, I don't want to do my data science work there. We don't even have a good native client for Jupyter notebooks yet, let alone for this new Jupyter alternative that doesn't support the existing Jupyter kernel protocol.
Not only that, but Pluto also apparently has some obnoxious UX limitations that remind me of other less-than-usable wannabe-Jupyter-notebooks (e.g. Apache Zeppelin, Databricks): https://towardsdatascience.com/could-pluto-be-a-real-jupyter...
In short: nice idea, but I'd rather see continued unification around Jupyter and a proper IDE that can at least emit and interact with Jupyter-compatible data.
On the other hand, the Jupyter notebook JSON format is bad for a variety of reasons (e.g. you need special tools for readable Git diffs) and I really wish we had all settled on R Markdown instead. But R has its own NIH tooling problem and nobody was ever going to adopt it because the R community itself (driven by RStudio) has little interest in sharing or interoperability with other languages.
</cynical-angry-rant>
[0]: https://blog.jupyter.org/nbterm-jupyter-notebooks-in-the-ter...
A good quality standalone "notebook editor" would be an incredible tool. Nteract exists, but is not "good".
Sure. VSCode with python and Jupyter extensions
Edit: Pluto uses Julia's package manager; moreover, Manifest.toml can be used to pin all of your project's dependencies so the notebook is repeatable, from a code perspective.
> julia -e "using Pluto; Pluto.run()"
Also, package manager can be used from inside Pluto. To install somethin, you can just write in a cell
> using Pkg
> Pkg.add("Package Name")
What does it even mean? What is a CLI interface for a programming language if not a REPL ?
I think these are two perfectly reasonable things to be annoyed by.
As someone who has never used Julia... Wow, that looks exceptionally painful compared to most other modern languages.
> What does it even mean? What is a CLI interface for a programming language if not a REPL ?
I guess they mean that the julia interpreter should be a good unix citizen (which is quite not at the moment). For example, while you can in theory create "julia scripts" by adding a julia shebang, this usage is not really well thought and has several friction points. Most notably, a very slow startup time, even of several seconds if you import some common packages. This makes said julia scripts essentially unusable.
The usual response of the julia community to these complaints is that "you are holding it wrong", and that you should use julia inside the proper REPL. Some people do not like this answer, and there's a tiny bit of drama around that.
One part of the ongoing effort to reduce latencies is to allow package authors to specify optimization levels on a per-module basis. This is great for plotting packages for example, since they usually don't benefit much from overly aggressive optimizations, so spending less time optimizing codes generally leads to a snappier experience. It is now even possible to opt into a module-specific fully interpreted mode, which can make a lot of sense for typical scripting tasks.
But if your point is the inability to do `julia script.jl` , yeah thats a pain point. Fortunately there has been some tooling to make running many jobs in a row easier: https://github.com/dmolina/DaemonMode.jl
I was also a bit harsh; you can at least do `julia -e 'using Pluto' -e 'Pluto.run()'`.
In contrast to AOT-compiled languages, PackageCompiler.jl doesn't statically analyze your code. So you need a "precompile script" that hopefully hits all callable methods (such a script will have to be made manually). The resulting "binary" is also massive.
sysimages are great, as is daemonmode. But really just do Revise at the REPL.
One pain point is that Rmarkdown uses a different pandoc installation when executed by Rstudio than from the terminal.
R was my first programming language and I got really spoiled with RStudio where everything "just works" and the "highlight code -> run in REPL" workflow is super smooth and tightly integrated. All I want is for that to work in other languages, but it seems like if you want it in Python you need to be running PyCharm or a similarly-heavyweight IDE (seriously, despite all the hype of VSCode there are still a ton of issues with just highlighting code and running it in an IPython terminal) and for Julia it just doesn't exist. If you really want a Jupyter-like workflow you can just use R Notebooks, which are literally just better in every way.
It sounds like you might have some good constructive criticism after trying several options. Care to elaborate on the shortcomings of various options?
I don’t like RStudio for the same reason I don’t like Matlab. I already have my editor and terminal workflow. I don’t want to use/learn a new tool for the privilege to use the language. Notebooks hit an acceptable middle ground where I can launch them via terminal. Notebooks have plenty of problems. Mainly, running cells out of order is just an incredibly dumb thing to be possible. This same problem is present in RStudio which you seem to enjoy (highlight and REPL) and you want it in other languages. If the code isn’t written to run in an order, a tool shouldn’t allow it.
I definitely agree that Python is a better general purpose computing language than R, but R's deployment story (i.e. packages) is much, much better than that of Python (pip/poetry/pipenv/conda/whatever came out this week). I honestly don't think that's the reason though, it's more that Python has much, much, much better developer mindshare.
Jupyter is a whole other world though, like iPython was the best thing ever as a proper REPL for python, and Jupyter was good for being able to do graphics with your code. That was all standard in the R world, with Sweave (which I wrote my thesis in), so it didn't appear to add a lot of value (to me, at least).
> I don’t like RStudio for the same reason I don’t like Matlab. I already have my editor and terminal workflow. I don’t want to use/learn a new tool for the privilege to use the language.
I am 100% with you on this, but Rstudio is just a nicer interface over the tools for literate programming in R, and the wonderfulness of Rmd vs ipynb is a thing of joy (to me, at least).
> Mainly, running cells out of order is just an incredibly dumb thing to be possible. This same problem is present in RStudio which you seem to enjoy (highlight and REPL) and you want it in other languages. If the code isn’t written to run in an order, a tool shouldn’t allow it.
So, this is a tricky one. I agree in principle, and I have a habit of continually re-running my documents to ensure that this doesn't cause problems, but there is definitely valid use-cases for out of order execution. Consider that you may often fit a model (which can take ages) and iterate on the visualisation/analysis code, but you don't want to re-run the modelling code every time you change a plot, which your solution would require.
Most of the tools claim to allow you to cache particular blocks, but I've never been able to get it to work reliably.
Unfortunately, it generally takes a couple of terrible situations before people learn the problems with this.
I do agree that this is the ideal though (As an example if Pluto is always reactive, then this workflow becomes much more difficult as when you change a downstream datapoint, the model will be re-run).
1) https://rstudio.github.io/reticulate/ 2) https://solutions.rstudio.com/python/ 3) https://blog.rstudio.com/2020/12/16/rstudio-connect-1-8-6-py... 4) https://ursalabs.org/
I really need to get a handle on ein, for when I'm inevitably dumped back into a notebook-driven environment (hard to avoid in DS these days).
Hear hear! A simple web-view inside a native application window is a huge improvement imho. If only JuptyerLab provided a simple interface to access menu elements as well, you could easily have a nearly complete native experience.
The closest we got was the org-mode file format with human-readable data for everything, but it seems tightly coupled with Emacs unless you only want to use it as Markdown replacement.
Personally I love org mode, but we'd need a jupyter plugin to convert ipynb to org and back to make it work.
Think of something you might use a spreadsheet for — Pluto has a similar feeling of instant feedback.
---
Some features that are missing:
– Some things are difficult to do with the keyboard; I used my mouse more than with other tools. The author doesn't like modal editing, but ideally they could be implemented with modifier keys (https://github.com/fonsp/Pluto.jl/issues/65)
- It's hard to understand what happens _within_ a cell — logging goes to the terminal rather than the notebook — and there aren't many introspection tools. This is an environment where transparency / introspection would be particularly helpful.
---
Pluto doesn't solve every problem, or completely replace notebooks; to respond to a couple of comments:
> I have many extremely long notebooks that would almost certainly crash if you tried to recompute the whole thing
Right, don't use Pluto for that! It's not one environment to rule them all
> Many of the cells won't work at all because the inputs are long gone
That seems bad! Pluto will help you ensure that doesn't happen.
There is talk about putting in execution barriers that would help with this, at the risk of making Pluto more complicated for users:
Execution barriers are a nice idea though. There is/was a Jupyter notebook extension for "initialization cells", but the whole notebook extension ecosystem seems kind of dead and it's unclear if Jupyter Lab will ever have equivalents.
@bind allow_run html"Run cell below <input type=checkbox>"
if allow_run
your_code
enddata -> model(data) -> output(model)
So if you go back to mess around with the data, your model and output could be or would be recomputed, which you would need to do eventually but not while making iterative tweaks.
Another commenter suggested adding checkboxes which is a good idea, although then you are managing a bunch of checkbox states.
On the other hand, not everyone remembers to re-run dependent cells. I’ve had many R notebooks handed in to me where an author didn’t check it runs top to bottom with fresh workspace.
So, if the system thinks it takes T seconds to compute a cell, it could only start recomputation after f(T) seconds without user input.
Finding a good function f is left as an exercise for the reader. That’s where good systems will add value. A good system likely would need a more complex f, which also has ideas about how much file and network I/O the steps take and whether steps can easily be cancelled.
Looking at the world’s best best reactive system, I think it never automatically fetches external data, and only recalculates stuff it knows it can cancel, and also has a decent idea about how much time each step will take.
Now that Excel has lambdas, that latter part has gone. You can write an Ackermann function in it (https://www.spreadsheetweb.com/how-to-calculate-the-ackerman...)
(Note that the UI for activating / deactivating the barrier will likely be changed before merging this PR)
Python and R (especially R) have plenty of libraries that are high-quality, or even industry standard, but which are decades old and feel it. Python's NLTK is 20 years old for example and it can feel grating switching between NLTK and spaCy. R has three different object systems (four according to some), so you might be using some ancient battle tested library with Hadley Wickham's latest cutting edge libraries.
For those interested in seeing Pluto in action I highly recommend checking out the course notebooks here: https://computationalthinking.mit.edu/Spring21/
Besides that, Pluto can bind UI elements to your code. You can make simple interactive games that run in Pluto! How it's not awesome?
Yet, trying Pluto, it seems to be outrageously slow and clunky. Is it expected? Sometimes it takes a few seconds to do something. I'm not talking about the initialization (which is still a shame, but that's a different issue). I'm talking about running individual cells with simple code. This is unusable as of today, at least on my 3-year old laptop.
EDIT: just running it on julia's "master" branch (v1.7.0-DEV), the initialization seems to be slower, but then the cells run maybe marginally faster. Looks good, but I could not push this to my students yet...
But yes, my tolerance towards this is higher as I am used to matlab.
You may like octave for that (my daily runner, that I want to replace with julia in the future). It recognizes the same language as matlab; but it is free software and, crucially, the startup time is negligible. For example, you can run octave scripts inside a bash loop, as if it was a calculator.
I also wish I had an excuse to get more into Julia. I really like Flux.
I've always thought this was the most annoying quirk of notebooks in general, so it's nice to see a different take.
front end programmers coding for data science use case <x3
It was much better.
If I wanted a software-grade, rock-solid data pipeline, I would just copy-paste some code from an existing notebook and run it on Papermill.
That sounds dangerous to me. If your computer crashes or you introduce a bug to your notebook, you could lose all that data. Personally, I prefer my notebooks to be reproducible at any point.
If you don't want to recompute dependent nodes, then use new names for your experiments rather than redefining old functions and variables. Yes, in some ways this is less convenient for you, but it's more convenient for people receiving your notebooks, that the notebook is always in a consistent state and reproducible.
Maybe it doesn't work well for your workflow, particularly if you're not sharing notebooks and keeping your notebooks small. On the other hand, if your workflow requires significant amounts of leaving notebooks in an inconsistent state, you may end up saving yourself significant frustration with larger notebooks and losing work due to losing track of your mental tracking of inconsistencies.
Also, if you hit a state that you really don't want to lose, you should probably do a quick git commit. You can always squash commits later if needed.
It might be worth changing your workflow, or it might not.
I think that's my main notebook gripe: they make it look like if you run the code you'll get these results, but that's not even close to the case. Many people abuse this. At this point, I pretty much assume anything in a Jupyter notebook isn't reproducible.
Otherwise, it should never been shared with other people or even contain relevant analysis you may need for yourself later.
But this is not enough - also the library dependencies need to be fixed. Pluto will make this very easy in the near-future: https://github.com/fonsp/Pluto.jl/pull/844
This is exactly what I want, always. In Jupyter I'm continuously doing the restart kernel and re-run all cells dance. It is annoying and I love another system optimized for that like Pluto, without those stupid non-deterministic cells.