Hacker News Reader

Comment by slashdev | Hacker News Reader

Correct. Python multi-threaded code is not magically thread-safe. What it did do is make C functions atomic, like dictionary and list methods, since they're implemented by C functions. The reason is the GIL is released and re-acquired while executing Python code (every 100 ops if memory serves), but isn't released by most C functions, making them called with the global lock held, which makes them atomic. You can still have a thread switch between any two Python op codes. e.g. foo += 1 may or may not be atomic depending on how many op codes it compiles to (IIRC, in CPython, it is not atomic.)

View on HN

One of the reasons I love using gevent (https://www.gevent.org) is that it's way of introducing concurrency that does preserve these atomicity guarantees! Broadly speaking, the only time your code will be interrupted by something else in the same process is if some function call yields control back to the event loop - so if your block is entirely made up of code you control that doesn't do any I/O, you can be sure it will run atomically. And you get it for free without needing to rewrite synchronous code into asyncio.

This does make me wonder, though, if gevent will survive PEP 703's massive changes to internal Python assumptions. That said, gevent does work on pypy, so there's some history with it being flexible enough to port. Hopefully it won't be left behind, otherwise codebases using gevent will see this as a Python 2 -> 3 level apocalypse!

I’ve used gevent, but I don’t remember this behavior. How does it prevent the GIL from being released when running Python code?

Ah - it's more that you wouldn't typically need to run multiple threads in the same process to handle concurrent requests. For instance, gunicorn used with gevent workers will typically fork processes to ensure all cores are used, but wouldn't require multiple threads per process - gevent would handle concurrency on a single OS thread within each process.

Ah, got it. That makes sense.

Very reasonable tradeoff for an application server, which is embarrassingly parallel and holds little state.

Same with asyncio, but you have explicit await to tell you where this happens.

But this only solves network concurrency problems.

As soon as you leave python, that C extension can give up the GIL and your other python threads can start running.

The GIL makes python code have cooperative threading. It does not protect from e.g. your thread's view of state mutating when you make a database call.

I also believe it is best practice not to mutate data without holding the GIL in extension code, not a requirement - but I have mucked with a lot of different extension API so I might be confused.

No, I wouldn’t call it cooperative threading, it’s still preemptive in that any Python thread can be switched with another at any instruction. That’s the same behavior as the operating system with the same potential for race conditions (except python instructions are higher level than machine code instructions.)

While C extensions can release the GIL, that only makes sense if they do enough work that a Python thread could get some things done in the meanwhile, and it wouldn’t be surprising to the caller. Obviously the C thread can’t interact with the Python world after the GIL has been released.

Pretty much. This makes it simpler to wrap C libraries and it is partly why Python became successful back in the day.

Reply on news.ycombinator.com