morebetterer

It would be inefficient to share data structures or interpreter internal state across threads. This would lead to the poor performance you described with mutex contention.

It is better to have an interpreter instance per thread, each having their own separate variable pool, and pass messages between the different interpreter instances. This model scales well with multicore systems and is faster than the multi-process equivalent.

It's also a simpler model to implement and maintain. Python's current multithreading model is too complicated from an implementation point of view.

Sounds a bit like a cross between the (now defunct) Google Sawzall project and the sqlite csvtable module:

https://github.com/softace/sqliteodbc/blob/master/csvtable.c

It's a shame the GIL won't be removed because it's perceived to be too difficult.

It's trivial to remove the GIL - probably a week of mechanical work. Just don't depend on global variables in the interpreter. No global state, no problem. But the Python C API has to be changed to always store interpreter state into a struct, and a pointer to that interpreter state struct has to be passed as the first argument in all C API calls. Not rocket science; this is what Lua does.

It's a political decision to keep the GIL, not a technical one. As for preserving C API backwards compatibility, it's a straw man argument - the Python API broke from 2.x to 3.x anyway. There's no such thing as "lesser breakage" - only breakage.