The nogil effort seems like such a better solution, that even if it breaks the C interface, subinterpreters aren't worth considering.
I will give it a shot.
Subinterpreters are better than multiple processes because:
- they have significantly less memory overhead
- they can move objects much faster between subinterpreters because they don’t need to serialize through a format like JSON
- since they are all in the same process you can implement things like atomics or channels easily.
Subinterpreter are better then no-Gil because:
- they make the code easier to reason about and debug relative to raw multi-theading
- they don’t negatively impact single threaded (basically all existing) python code performance
- they don’t require any changes to the C interface, preventing a fractured ecosystem
- they can’t have data races
I think this deserves an extra callout because even your multi-threaded Python programs are effectively single threaded and benefiting from the performance gain.
Why do you think that you would need to serialize to JSON? Pipes and sockets can deal with binary data just fine. With shared memory, there wouldn't be any difference at all.
> - since they are all in the same process you can implement things like atomics or channels easily.
This is also possible with shared memory.
AFAICT the advantage of subinterpreters over subprocesses are:
- lower memory overhead
- faster creation/destruction time
- ability to share global data (with subprocesses the data would either need to be duplicated or live in shared memory)
But you do bring up some good points for ways you could achieve similar goals without the need to make the interpreters thread safe.
There is multiprocessing.Queue (https://docs.python.org/3/library/multiprocessing.html#multi...).
I don't know if it uses shared memory, or rather sockets or pipes, but this is just an implementation detail.
My point is that there is no fundamental difference between isolated interpreters and processes when it comes to data sharing. Either way, you need a (binary) serialization format and some thread/process-safe queue.
I would have naively assumed that you could repurpose multiprocessing.Queue for passing data between multiple interpreters; you would just need to replace the underlying communication mechanism (sockets, pipes, shared memory, whatever it is) with a queue + mutex. But then again, I'm not familiar with the actual code base. If there are any complications that I didn't take into acccount, I would be curious to hear about them.
Interestingly, the PEP authors currently don't propose an API for exchanging data and instead suggest using raw pipes:
> https://peps.python.org/pep-0554/#api-for-sharing-data
Of course, this is just a temporary hack. It would be ridiculous to use actual pipes for sharing data within the same process...
How so? Asking out of curiosity.
Which isn't really different than a similar pattern of isolation in threading or multiprocessing.
That would be a huge advantage, but it's not there yet. According to PEP 554 [1] the only mechanism for sharing data is sending bytes through OS pipes, which is exactly the same as for multiprocessing and requires the same sort of serialization.
https://docs.python.org/3/library/multiprocessing.html#multi...
In a message passing system there’s always going to need to be some form of serialization. I’ll wager that pickle is fast and flexible enough for most cases and for those that aren’t, using something like flatbuffers or capn proto in shared memory wouldn’t be too much of a lift to integrate.
Although all of that has long been possible in a multiple-process architecture, so I’m also curious to know if there are any real advantages to subinterpreters. From this message [1] linked to from the PEP it sounds like the author once thought that object sharing was a possibility, but if it’s not there seem to be no real benefits over multiprocessing and one big downside (the GIL).
Contrast with ruby’s Ractor system [2], which is similar to the subinterpreter concept but allows true parallelism within a single process by giving each ractor its own interpreter lock, along with a system for marking an object as immutable so it can be shared among ractors.
[1] https://mail.python.org/pipermail/python-ideas/2017-Septembe...
- They absolutely do have to serialize, usually via pickle. I'm pretty sure objects are not sharable between subinterpreters and there is not a plan for that. The main reason people think subinterpreters are good ("you can just share the memory!") is not actually true.
- They don't require any changes to the C interface because those changes were already made, and a fair amount of cost was paid by C library maintainers. So it's true, subinterpreters are at an advantage in this regard, but that's more of a political question than a technical one
The hidden really hard problem is that extension modules may have been written relying on GIL behavior for thread safety... these may be undocumented and unmaintained.
Even so I hope the community decides it is worth it. A glue language with actual MT support would be much more useful.
> Even if it breaks the c interface
Than most of your python packages wouldn't work. A python that isn't backwards compatible? Ya, that has been tried once before, and was a disaster. If you want a non-backwards compatible gil-less python, it already exists. You can find versions of nogil online.
I suspect sub-interpreters are a punt and a feint.
My guess is that there will likely be exactly 2 sub-interpreters in most Python code. One which talks to an old C API with a GIL and one which talks to a new C API without a GIL.
It's going to be a lot easier to manage handing objects between two Python sub-interpreters than to manage handing objects between two incompatible ABIs.