Calling Rust from Python | Hacker News Reader

Calling Rust from Python

blog.frankel.ch

113 pointsRebootStr32 comments

I would not recommend FFI + ctypes. Maintaining the bindings is tedious and error-prone. Also, Rust FFI/unsafe can be tricky even for experienced Rust devs.

Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.

[1] https://github.com/PyO3/pyo3

Right now FFI + ctypes has one big advantage: it supports true parallelism with GIL-per-subinterpreter in Python 3.12. AFAIK all the higher-level binding libraries/tools (PyO3, pybind11, nanobind, Cython) don't support this yet; and in fact can't really support it without API-breaking fundamental design changes.

But wasn't the whole point to drop down to a lower language, release the GIL and then do parallel processing? Gil per subinterpretee makes things easy python side but I'm not sure how much of it's relevant if you are using something like pyO3

Do you know what kind of design changes would be needed?

I think you are right. But as a proof of concept is an interesting read. I've work extensively with Rust and PyO3 and Maturin and even for a medium to large project is great. I wrote a bit about that here:

https://www.nhatcher.com/post/rust-in-anger/

I'd also recommend PyO3 and Maturin. The amount of help these crates give is mind boggling (automatic type marshaling and even github CI jobs for creating cross platform precompiled wheels).

I've created a library using this crate (https://github.com/wiktor-k/pysequoia/) and most of the time I could just focus on the problem domain instead of technical details of the bindings.

There are just a couple of smaller issues (eg. Python to Rust async is not built in) but overall it's really nice.

Agreed. It's also instructive to look at how C extensions are written for the CPython.

I wouldn't necessarily expect a demo; but not mentioning it at all seems like it could lead readers down a path where they choose the "most performant" option presented (FFI + ctypes). The post has a "To go further:" link section where even a quick link to PyO3 would go a long way. So I thought I'd mention the downsides and an alternative on HN.

Absolutely, I was surprised anyone would recommend ctypes for Python/Rust today. I've written a few Python libraries using Rust to do the slow parts, and PyO3/maturin is a great experience.

Even a decade ago I was seeing projects stop using ctypes and instead use things like cffi. Honestly I couldn't tell you exactly why ctypes is so bad but I do know that cffi integrates a little closer with the c compiler and has better header support.

Some related past threads:

How to write Python extensions in Rust with PyO3 - https://news.ycombinator.com/item?id=34968186 - Feb 2023 (9 comments)

PyO3 – Python Extensions in Rust - https://news.ycombinator.com/item?id=32247452 - July 2022 (1 comment)

Calling Rust from Python using PyO3 - https://news.ycombinator.com/item?id=29368530 - Nov 2021 (49 comments)

PyO3: Rust Bindings for the Python Interpreter - https://news.ycombinator.com/item?id=25956502 - Jan 2021 (77 comments)

Writing Python Extensions in Rust Using PyO3 - https://news.ycombinator.com/item?id=17423013 - June 2018 (1 comment)

PyO3: Python Rust binding - https://news.ycombinator.com/item?id=14859844 - July 2017 (15 comments)

(First Release) PyO3: a Python-Rust Binding Library - https://news.ycombinator.com/item?id=14846606 - July 2017 (1 comment)

> I would not recommend FFI + ctypes.

Well it depends. C is the lingua franca of Computer Science. These days I write a lot of code that needs to be accessible to a variety of languages and environments. Python, C#, C++, Matlab, Unity, Unreal, etc.

Write a C API and your code can be used anywhere. Now if you know you only ever need to support Python then sure look into PyO3. But if you have mixed use you can’t be the flexibility of a C API.

Honestly though, I would likely still choose to implement the logic in Rust as a crate/library in a workspace, the (C)Python extensions as a PyO3 dylib/separate crate, and a C FFI dylib as yet another crate.

The benefit is that you only need to ship the CPython extension, you can build it with cargo, and everything's in one place. I've maintained bindings for Python and C# to native libraries, and it's usually a pain for various reasons.

The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.

> The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.

Definitely true. That said I think there is an under appreciated beautiy to a clean C API.

It’s easy to “upgrade” a C API with per-language/environment bindings. It can be near impossible to “downgrade” a high-level API to C.

And Rust just uses the C ABI, so wouldn’t your comments include “in Rust”?

Just looking at the function signatures there:

   fn string_sum(_py: Python<'_>, m: &PyModule)

is the idea that the `Python` type represents the GIL and you're actually acquiring it from within your Rust application?

Edit: Yes it does, as ynik rightly points out, and documented here https://pyo3.rs/v0.19.2/types#gil-lifetimes-mutability-and-p... or here https://docs.rs/pyo3/0.19.2/pyo3/marker/struct.Python.html#o...

The Python<'_> type appearing as a parameter means the caller has already acquired the GIL for you. The GIL is held when Python calls into extension modules, and PyO3 doesn't automatically release it (but you can do so yourself).

It's essentially a dummy parameter that serves as proof that the GIL is held. You need to provide it to PyO3 when calling back into the Python runtime, that way the compiler ensures the GIL is still held where required.

Isn't PyO3 just a set of macros and helpers around FFI and ctypes?

"just" undersells it a bit, mainly around type conversions.

Also, there is no ctypes involved, as the result is a CPython extension that just loads on the Python side.

In the same sense that rust is just a set of macros and helpers around assembly.

Also no ctypes.

PyO3 is great. I have been using it to rewrite some slow python code. My only complaint is that documentation is a bit rough. For someone who understands python, but not rust, there was a pretty big hill to climb to understand type conversions.

There are also things I am still struggling with. For example, there is a rust function I want to repeatedly call from python, that takes in a large unchanging python array. How can I pass the array from python to rust only once, i.e do the type conversion just once, so I don't have to pay that cost repeatedly.

I recommend creating a pyclass over a struct that stores the converted data. Instantiate it once and then repeatedly call a function on it in a pymethods impl with the parameters that differ.

FFI+cffi has the advantage that you leave Python specifics out of your Rust binding, so it's effectively a proper C binding that can be used for other FFI integrations as well.

I have used PyO3 for my Rust code to calling into a Python SDK api. It works like a charm.

Polars uses PyO3 too, looks quite simple.

Small note for the author: I was a bit confused at first by the use of "Nix-based" to mean POSIX-compliant [1] since Nix is the name of a package manager that NixOS is based on. I've only seen "*nix" to refer to Unix-like systems.

[1] Which seems like the right condition here? since I believe Unix domain sockets are required by POSIX.

Duly noted

Also, I forgot to say: thanks for the submission!

As others have pointed out: none of these three “generic” methods is appropriate when a more precise one (such as dedicated bindings between a pair of languages) exists.

PyO3 is practical evidence of Rust’s claims around holistic security: a user can write safe Rust and safe Python without having to directly unsafely traverse the C ABI between them, or take a potentially unacceptable performance hit from RPC or IPC.

I read https://tratt.net/laurie/blog/2016/fine_grained_language_com... this week, and now I’m obsessed with how all three of the techniques presented here suck, and we should just be able to mix languages freely.

check out uniffi for a complete workflow for FFI in many languages: https://github.com/mozilla/uniffi-rs

as long as you don't need to cross compile pyO3 makes calling python from rust in ffi extremely simple and convenient as it's abstracts away so the ffi, ctypes parts

Reply on news.ycombinator.com