Hacker News Reader

Comment by qprofyeh | Hacker News Reader

Ergonomic multi-core multithreading is already solved, for me at least, with ProcessPoolExecutor.map(). I wonder what advantages this direction brings to the table.

View on HN

You will now be able to, in 3.12 run multiple CPython VMs inside the same process w/o having them share state. Previously CPython VM state was spread over a bunch of global variables so you had one GIL per process embedding Python.

For your usecase, eventually that ProcessPoolExecutor.map will be GillFreeThreadPoolExecutor.map and all the cross process serialization shenanigans will go away.

But instead of cross-process serialization we get cross-interpreter serialization, so serialization shenanigans don't go away.

There is support in the newest pickle protocol for using a shared buffer to transfer data more efficiently, but that would work in multiprocessing [0] just as well as in subinterpreters (and currently isn't implemented in either one).

[0] https://github.com/python/cpython/issues/89467

I agree in the short term, those problems aren't entirely solved, but it does provide for an eventual serialization free way to communicate between threads.

Queues, immutable records, atomic refcounts, a global object heap for shared items. There are lots of way forward here that don't involve a full SERDES round trip.

Not needing multiple processes and not having to use slow IPC with serialization for them to talk to one another.

That's not multithreading

> is already solved

Has been for decades outside python

Reply on news.ycombinator.com