Real Multithreading Is Coming to Python

martinheinz.dev

43 points·nalgeon··4 comments

Important to note that this is different from the nogil project ( https://github.com/colesbury/nogil ) - this is about the feature that's landing soon in Python 3.12 which will allow multiple Python sub-interpreters each with their own GIL - hence enabling Python code to run multi-threaded in the same process, albeit with a fair amount of extra work to co-ordinate between those different interpreters.

The tutorial itself is fantastic - it's not an easy feature to try out right now but I got there by following the step-by-step instructions.

Also interesting is this project, linked to from the end of the tutorial: https://github.com/jsbueno/extrainterpreters

Posted my own set of short notes on my blog: https://simonwillison.net/2023/May/15/per-interpreter-gils/

This looks interesting, eagerly looking forward to learning what IPC will look like compared to python multiprocessing, Golang goroutines, C++ threads, and Java threads.

I wish there were a way to do away with the GIL entirely ^^.

https://github.com/colesbury/nogil does manage to get rid of the GIL, but it's not certain to make it into Python core. The main problem is the amount of existing libraries that depend on the existence of the GIL without realizing it - breaking those would be extremely disruptive.

On the other hand, it would fix cases where there are somewhat rare hangs that are hard to reproduce, for example in multithreaded applications embedding Python. A while ago, I hit a deadlock in LibreOffice's Python extension system. I managed to fix it by changing the GIL to the more modern Python API for that, but I never managed to produce a reasonably small test case, so I didn't submit it.

Reply on news.ycombinator.com