How to write Python extensions in Rust with PyO3 | Hacker News Reader

How to write Python extensions in Rust with PyO3

infoworld.com

45 pointsjeremychone9 comments

PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs

It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust.

This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.

Pola.rs is another good example of how a great Rust library can have an excellent Python façade. Polars ingenuously uses Python's operator overloading scheme to maximize data frames operation efficiencies.

I think Python & Rust can become great pairs, especially in the data processing field.

I found PyO3 to be a bit weird to use. There's plenty of documentation, but it's kind of vague on the general basic concepts, which makes the library hard to understand (even though I have a background in writing CPython extensions in C).

Also, once you play with it more than 30m, you quickly realize that struct attributes are copied all over the place... which is not only slow, but more sadly any mutable function on an attribute will just silently modify a temporary copy...

Isn’t python “memory safe” too?

From my experience the trickiest bit of getting python extensions to work (i.e. not segfault) is dealing with the lifetimes of the objects in the wrapped library. If python owns the object you’re golden, if it borrows a reference then things get real complicated real fast. It usually takes a lot longer to get the ownership model correct than doing the grunt work of wrapping the library — unless you own the lib and make it python friendly it’s usually a case of “do what you can with what you were given”.

Blender’s python API is full of cases where the python object can’t be trusted to be valid if you do anything with the underlying container and it is super easy to crash the whole program doing something that should Just Work™.

Wondering how rust would deal with the cross-domain model where nothing can be guaranteed once python gets its dirty little paws on the pointer?

I assume PyO3 is a reference to the chemical formula of iron oxide (a.k.a. rust), which is Fe2O3?

I think it's a reference to the -o3 optimalization level in gcc.

PyO3 works great. I've written an example repository for a hello world app for this if anyone's interested: https://github.com/russnes/rust_python_package_example

For a somewhat more "productive" hello world (as in a very simple application which does useful things), there's nh3 (https://github.com/messense/nh3) which is a relatively small wrapper around ammonia (https://github.com/rust-ammonia/ammonia), an HTML sanitisation library.

The lib.rs is all of 125 SLOC (and about half of that is owing to the handling of `attribute_filter` which is pretty demanding on the glue layer: https://github.com/messense/nh3/commit/72be3e6728b7ceb9185e7...).

Together with rust-numpy, it is a great way of writing numeric code (using the ndarray library) https://github.com/PyO3/rust-numpy

Reply on news.ycombinator.com