Comment by morebetterer | Hacker News Reader

Comment by morebetterer | Hacker News Reader

Parent Full thread

It's a shame the GIL won't be removed because it's perceived to be too difficult.

It's trivial to remove the GIL - probably a week of mechanical work. Just don't depend on global variables in the interpreter. No global state, no problem. But the Python C API has to be changed to always store interpreter state into a struct, and a pointer to that interpreter state struct has to be passed as the first argument in all C API calls. Not rocket science; this is what Lua does.

It's a political decision to keep the GIL, not a technical one. As for preserving C API backwards compatibility, it's a straw man argument - the Python API broke from 2.x to 3.x anyway. There's no such thing as "lesser breakage" - only breakage.

Supporting multiple interpreters in a single process is not what most people mean when they talk about removing the GIL. Objects from one interpreter could not be used safely from another interpreter. This would be handy in a few situations but is essentially not all that different to multiprocessing.

What people usually mean when they talk about removing the GIL is having multi-threaded code make use of multiple cpu cores (as it does in Jython.) This would involved splitting the GIL into more fine-grained locks. Unfortunately experiments taking this approach have so far shown significant performance impacts for single threaded code.

It would be inefficient to share data structures or interpreter internal state across threads. This would lead to the poor performance you described with mutex contention.

It is better to have an interpreter instance per thread, each having their own separate variable pool, and pass messages between the different interpreter instances. This model scales well with multicore systems and is faster than the multi-process equivalent.

It's also a simpler model to implement and maintain. Python's current multithreading model is too complicated from an implementation point of view.

Reply on news.ycombinator.com