Hacker News Reader

Comment by guitarbill | Hacker News Reader

I would not recommend FFI + ctypes. Maintaining the bindings is tedious and error-prone. Also, Rust FFI/unsafe can be tricky even for experienced Rust devs.

Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.

[1] https://github.com/PyO3/pyo3

View on HN

Right now FFI + ctypes has one big advantage: it supports true parallelism with GIL-per-subinterpreter in Python 3.12. AFAIK all the higher-level binding libraries/tools (PyO3, pybind11, nanobind, Cython) don't support this yet; and in fact can't really support it without API-breaking fundamental design changes.

Do you know what kind of design changes would be needed?

But wasn't the whole point to drop down to a lower language, release the GIL and then do parallel processing? Gil per subinterpretee makes things easy python side but I'm not sure how much of it's relevant if you are using something like pyO3

I think you are right. But as a proof of concept is an interesting read. I've work extensively with Rust and PyO3 and Maturin and even for a medium to large project is great. I wrote a bit about that here:

https://www.nhatcher.com/post/rust-in-anger/

I'd also recommend PyO3 and Maturin. The amount of help these crates give is mind boggling (automatic type marshaling and even github CI jobs for creating cross platform precompiled wheels).

I've created a library using this crate (https://github.com/wiktor-k/pysequoia/) and most of the time I could just focus on the problem domain instead of technical details of the bindings.

There are just a couple of smaller issues (eg. Python to Rust async is not built in) but overall it's really nice.

Agreed. It's also instructive to look at how C extensions are written for the CPython.

I wouldn't necessarily expect a demo; but not mentioning it at all seems like it could lead readers down a path where they choose the "most performant" option presented (FFI + ctypes). The post has a "To go further:" link section where even a quick link to PyO3 would go a long way. So I thought I'd mention the downsides and an alternative on HN.

Absolutely, I was surprised anyone would recommend ctypes for Python/Rust today. I've written a few Python libraries using Rust to do the slow parts, and PyO3/maturin is a great experience.

Even a decade ago I was seeing projects stop using ctypes and instead use things like cffi. Honestly I couldn't tell you exactly why ctypes is so bad but I do know that cffi integrates a little closer with the c compiler and has better header support.

Some related past threads:

How to write Python extensions in Rust with PyO3 - https://news.ycombinator.com/item?id=34968186 - Feb 2023 (9 comments)

PyO3 – Python Extensions in Rust - https://news.ycombinator.com/item?id=32247452 - July 2022 (1 comment)

Calling Rust from Python using PyO3 - https://news.ycombinator.com/item?id=29368530 - Nov 2021 (49 comments)

PyO3: Rust Bindings for the Python Interpreter - https://news.ycombinator.com/item?id=25956502 - Jan 2021 (77 comments)

Writing Python Extensions in Rust Using PyO3 - https://news.ycombinator.com/item?id=17423013 - June 2018 (1 comment)

PyO3: Python Rust binding - https://news.ycombinator.com/item?id=14859844 - July 2017 (15 comments)

(First Release) PyO3: a Python-Rust Binding Library - https://news.ycombinator.com/item?id=14846606 - July 2017 (1 comment)

> I would not recommend FFI + ctypes.

Well it depends. C is the lingua franca of Computer Science. These days I write a lot of code that needs to be accessible to a variety of languages and environments. Python, C#, C++, Matlab, Unity, Unreal, etc.

Write a C API and your code can be used anywhere. Now if you know you only ever need to support Python then sure look into PyO3. But if you have mixed use you can’t be the flexibility of a C API.

Honestly though, I would likely still choose to implement the logic in Rust as a crate/library in a workspace, the (C)Python extensions as a PyO3 dylib/separate crate, and a C FFI dylib as yet another crate.

The benefit is that you only need to ship the CPython extension, you can build it with cargo, and everything's in one place. I've maintained bindings for Python and C# to native libraries, and it's usually a pain for various reasons.

The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.

> The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.

Definitely true. That said I think there is an under appreciated beautiy to a clean C API.

It’s easy to “upgrade” a C API with per-language/environment bindings. It can be near impossible to “downgrade” a high-level API to C.

And Rust just uses the C ABI, so wouldn’t your comments include “in Rust”?

Just looking at the function signatures there:

   fn string_sum(_py: Python<'_>, m: &PyModule)

is the idea that the `Python` type represents the GIL and you're actually acquiring it from within your Rust application?

Edit: Yes it does, as ynik rightly points out, and documented here https://pyo3.rs/v0.19.2/types#gil-lifetimes-mutability-and-p... or here https://docs.rs/pyo3/0.19.2/pyo3/marker/struct.Python.html#o...

The Python<'_> type appearing as a parameter means the caller has already acquired the GIL for you. The GIL is held when Python calls into extension modules, and PyO3 doesn't automatically release it (but you can do so yourself).

It's essentially a dummy parameter that serves as proof that the GIL is held. You need to provide it to PyO3 when calling back into the Python runtime, that way the compiler ensures the GIL is still held where required.

Isn't PyO3 just a set of macros and helpers around FFI and ctypes?

"just" undersells it a bit, mainly around type conversions.

Also, there is no ctypes involved, as the result is a CPython extension that just loads on the Python side.

In the same sense that rust is just a set of macros and helpers around assembly.

Also no ctypes.

PyO3 is great. I have been using it to rewrite some slow python code. My only complaint is that documentation is a bit rough. For someone who understands python, but not rust, there was a pretty big hill to climb to understand type conversions.

There are also things I am still struggling with. For example, there is a rust function I want to repeatedly call from python, that takes in a large unchanging python array. How can I pass the array from python to rust only once, i.e do the type conversion just once, so I don't have to pay that cost repeatedly.

I recommend creating a pyclass over a struct that stores the converted data. Instantiate it once and then repeatedly call a function on it in a pymethods impl with the parameters that differ.

FFI+cffi has the advantage that you leave Python specifics out of your Rust binding, so it's effectively a proper C binding that can be used for other FFI integrations as well.

I have used PyO3 for my Rust code to calling into a Python SDK api. It works like a charm.

Polars uses PyO3 too, looks quite simple.

Reply on news.ycombinator.com