Hacker News Reader

Comment by zokier | Hacker News Reader

> If we have a Python buffer, and we want to represent the data in Rust, how should we do so? The natural answer would be a &[u8], but as you may have picked up, in the face of the possibility of concurrent writes, this would be unsound. Similarly, an &mut [u8] is unacceptable because the Python buffer protocol provides no assurances that only one mutable buffer is handed out at a time.

Idk, to me the natural answer for anything crossing "ffi" border would be *mut u8. Afaik rustc makes fewer assumptions about raw pointers and it might be closer to sound interface?

View on HN

Yeah a raw pointer makes sense but is equally unhelpful for the caller, since there's still no safe way to access the data.

It sounds to me like the Python buffer protocol is under-specified, which can lead to data races in native code.

Here's an example of how this could occur without Rust being involved:

- Two python threads with a shared buffer object

- One thread instructs a C extension to write into the buffer

- The other thread instructs a C extension to read from the buffer

- The buffer is implemented to return the same pointer for reads and writes (not forbidden by the spec https://docs.python.org/3/c-api/buffer.html#buffer-request-t...)

- The C extension releases the GIL while accessing the buffer (explicitly allowed by the spec)

In this case, the C extension is reading and writing to the same buffer at the same time without the use of atomics, which is the definition of a data race, and therefore triggers UB.

IMO, the best fix would be for the buffer implementation to expose a "concurrency mode", and require that users of the buffer API use the concurrency mode specified by the buffer when accessing it. In one mode, no synchronization is necessary because the buffer only ever hands out one writable pointer or multiple readable pointers but never both. In another mode, you might be required to have the GIL, in another mode there might be a buffer-specific RWLock you have to hold while accessing it.

"Yeah a raw pointer makes sense but is equally unhelpful for the caller, since there's still no safe way to access the data."

The problem is exactly that there is no safe way to access the data, according to Rust's definition of safety. The entire point of Rust is to exclude the C-esque "well, I have a pointer, I guess I have the right to do whatever I want with it, I hope, maybe?" access model, but that's what Python is operating under. No amount of Rust hackery can add Rust's promises onto the Python code. I'd argue that the "correct" answer is to hand back an opaque pointer value into the Rust, and obligate all Rust access to that opaque pointer to be through "unsafe". To put things back into Rust's safe space would require an unsafe copy, but at least the copy would be back in "true" Rust.

The problem here isn't that that is incorrect, the problem is that it's uselessly unergonomic, both for the programmer, and because I may not want to pay the price of making a copy. At this point we're just down to figuring out what promise of Rust we want to break. (The only reason we're not discussing which promises of Python we want to break is that on this dimension, it doesn't make any. There are other conceivable situations where we might have compromise in both directions, e.g., Haskell's immutability.) We have the freedom of acknowledging there isn't a perfect solution, and then pursuing the "good enough"... and the "good enough" is probably to just hand Rust the buffer wrapped in whatever Rust type you feel like/is most appropriate from Rust's point of view, and then just saying "all this type can do, and therefore all it does, is constrain my Rust code; I can do nothing about the Python at all on this level". Since there is no solution, spinning about trying to find one is not terribly productive.

Reply on news.ycombinator.com