So besides the unproven possibility of removing the GIL, subinterpreters are the best way forward, better than threads or the multiprocessing package.
So besides the unproven possibility of removing the GIL, subinterpreters are the best way forward, better than threads or the multiprocessing package.
That's not accurate.
> they have their place but aren't related to parallelisation
You can parallelize all sorts of things with Python threads--just not some things you'd expect to be able to parallelize, due to the GIL. Waiting on or buffering I/O, calling out to compiled code, doing cryptographic operations--all of those can be parallelized, as (in many cases) they entail releasing the GIL.
> But true multiprocessing is ugly when you have hundreds of cores
Why?
> There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever.
I have no idea what this means. What do UIs have to do with process groups? Do you know how many processes your Chrome instance is running on the operating system? There are very solid conventions regarding process management, at least on Unix-ish systems: process groups and parent-child relationships are well established and well understood, as is their relationship with signals and signal handling.
Green threads also allow for parallelism, if they're scheduled onto more than one executor.
I think threads are a bad abstraction for doing parallelism personally, though. Programs designed to run in parallel should be deterministic, unless they need concurrency for some other reason. I think trying to shoehorn parallel programming into Python's threads isn't necessarily the best approach.
As far as I'm concerned, if I'm using threads and they happen to get scheduled on to multiple cores, then that's a nice optimization, but isn't necessary for what I use threads for.