Evcxr: A Rust REPL and Jupyter Kernel | Hacker News Reader

Evcxr: A Rust REPL and Jupyter Kernel

github.com

170 pointsbatterylow27 comments

There's an explanation of how it works: https://github.com/google/evcxr/blob/master/evcxr/HOW_IT_WOR...

That's an impressive amount of hacks under the hood! I thought maybe there would be an interpreter involved, but it really does invoke rustc and then loads the code with dynamic linking.

If anyone's interested in a collection of scientific programming/machine learning libraries written in Rust, we maintain a collection of them across different domains at https://www.arewelearningyet.com/.

An example Rust Jupyter Notebook for data visualization using this tool: https://plotters-rs.github.io/plotters-doc-data/evcxr-jupyte...

Rust hold interesting promise for data analysis and visualization at scale, especially as someone who hits use cases where Python/R start to bottleneck. The ecosystem is not there yet, though.

not to sound like a broken record, but Julia has the performance while also having dynamic memory management and an ecosystem close to Python/R (depending on the area).

Rust is nice when you want something lightweight and fast, since Julia has a pretty heavyweight runtime.

Julia is cool, but can I have a version where arrays start at zero?

No need for another version, Julia supports custom indices by default. Check out https://docs.julialang.org/en/v1/devdocs/offset-arrays/ and https://github.com/JuliaArrays/OffsetArrays.jl

Yes, Julia looks very impressive, a language I'd like to try at some point.

"Lightweight and fast" are the key features I'm looking for.

To play an equally broken record: Have julia's ridiculously long precompilation times improved recently?

Yes, absolutely. Version 1.5 brought significantly faster precompilation and version 1.6 (currently in beta) brings some precompilation speed improvements on top of that as well as multithreaded precompilation, so things have improved a ton in that regard.

We're expecting more benefits down the pipe as well with better AOT compilation support so you don't have to redo compilation so much between sessions.

Compilation and precompilation latency have been the two most consistent focuses of the developer team over the past couple years.

Thanks, looks like I should give julia a try again.

Version 1.6 (beta) is significantly faster (5-10x on my machine) than 1.5, largely due to code invalidations and parallelism. And 1.5 is faster (in both runtime and compiletime) than earlier versions.

I also imagine future iterations will focus on AOT options as well (we already have some in the form of PackageCompiler.jl),

I enjoy literate programming so Evcxr has made learning Rust a familiar experience.

The data science ecosystem isn’t quite there yet, but you can still do some things easily if you want to.

If you’re interested, you may want to check out my book on the topic https://datacrayon.com/shop/product/data-analysis-with-rust-...

I have been using evcxr for the last 2-3 weeks and I'm loving it. There are two things that bother me: (a) fancy UI components seem to only work with Python and C++ kernel (not theoretically, but with available packages), (b) while you can redefine values and functions when you're iterating, you cannot do so with structs (yes, big structs go to a local package that I depend on, but sometimes I have ad-hoc utility structs).

Here's the xeus-cling (Jupyter C++ Kernel) source: https://github.com/jupyter-xeus/xeus-cling/tree/master/src

Do any of the other non-Python Jupyter kernels have examples of working fancy UI components? https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Jupyter kernels implement the Jupyter kernel message spec. Introspection, Completion: https://jupyter-client.readthedocs.io/en/latest/messaging.ht...

Debugging (w/ DAP: Debug Adapter Protocol) https://jupyter-client.readthedocs.io/en/latest/messaging.ht...

A `display_data` Jupyter kernel message includes a `data` key with a dict value: "The data dict contains key/value pairs, where the keys are MIME types and the values are the raw data of the representation in that format." https://jupyter-client.readthedocs.io/en/latest/messaging.ht...

This looks like it does something with MIME bundles: https://github.com/jupyter-xeus/xeus-cling/blob/00b1fa69d17b...

ipython.display: https://github.com/ipython/ipython/blob/master/IPython/displ...

ipython.core.display: https://github.com/ipython/ipython/blob/master/IPython/core/...

ipython.lib.display: https://github.com/ipython/ipython/blob/master/IPython/lib/d...

You can also run Jupyter kernels in a shell with jupyter/jupyter_console:

    pip install jupyter-console jupyter-client
    jupyter kernelspec list
    jupyter console --kernel python3

I added jupyter notebook support to my nsi[1] crate via evcxr.

It allows you to do a 3D (offline) render with 3Delight (on AWS, if you want) – directly into a notebook.

Here is an example notebook[2].

I haven't managed to update the image dynamically while it's rendering (there's no support for that in evcxr). If anyone knows how to do this, I'd be keen to add that too.

[1] https://crates.io/crates/nsi

[2] https://github.com/virtualritz/nsi/blob/master/examples/jupy...

Are you talking about streaming data from a thread on the backend to say a canvas or image element in the browser?

If you streaming in blocks on the server side and compositing them into a final render, in this case there are a couple hacks.

1) just use a timer and reload the image every x seconds

2) do a hanging read on and endpoint, return updated image. This has the nice quality of the server being able to decide the framerate.

Yes, I am streaming buckets/blocks. That would be both on the evcxr side, right?

I was wondering if sending it as an MNG with loop=1 would work.

Rich display handles sending html/svg/etc back to the UI layer.

https://ipython.readthedocs.io/en/stable/config/integrating....

You might be able to do everything in there with a little JS.

I used this about a year ago for a small astrophysics simulation+plotting etc, together with futhark for some GPU code. It wasn't perfect (some crashes and weird behaviour), but worked remarkably well. I imagine it's even better today.

It was really useful to have easy acces to the simulation code and datastructures from the notebook where I made the plots, and use the same language for both. Glad to see the project still going strong.

This is amazing! But I still posit there are better UI paradigms than Jupyter (and REPL). Working in Jupyter is like having one hand tied behind my back.

I'm interested in what better/ other UI paradigms there are. Could you elaborate?

Not OP, but I like to avoid Jupyter where I can too. I like using a REPL though.

For very interactive, experimental work I like having code in one pane and a REPL in another separate one - so that you can highlight and send snippets of a file to the REPL with a keystroke or two, and the code and results aren't mixed together.

RStudio or Vim Slime[1] would be my favourite examples of this kind of approach.

For more "static" literate programming, I prefer the way RMarkdown and Sweave work to Jupyter Notebooks (though I don't even use R anymore). Both have their strengths and weaknesses of course.

[1] https://github.com/jpalardy/vim-slime

Despite my aversion to these 'live-coding' environments for compiled languages, I'm actually interested to see what can be managed with Rust. The dynamic linking they are using sort of reminds me of rust-analyzer which quickly re-parses it's tree at every save. I don't code in a way that normally requires interpretative feedback but quality tools like evcxr might contribute to changing my habits.

First and third party materials to get up and running with Evcxr here https://github.com/google/evcxr/blob/master/evcxr_jupyter/RE...

I really loved the Rails console for developing web apps. Write a few lines on the REPL, then past them in an app/model file.

Knowing Rust (and other strong typed langs) I understand that it is not so easy to implement this workflow for these languages. But looking at this project I kind of have new hopes. With the strong languages it should even be possible to have intellisense-like completion/suggestion available on the REPL!

Would it be, eventually, possible use Evcxr for this purpose?

wow, this looks so cool, nice job!

Reply on news.ycombinator.com