Comment by std_throwawayay | Hacker News Reader

Comment by std_throwawayay | Hacker News Reader

Parent Full thread

std_throwawayay

Right now you have one mutex for everything (the GIL itself) and everything else doesn't need locking. In order to achieve similar convenience without the GIL you would have to trade this for one mutex for every single data structure. Because the data structures in python are so flexible, every single variable needs its own mutex then. Locking every single variable access would be enormously costly.

Other languages achieve a good compromise by clustering data structures into fewer parts with only a handful of mutexes that are locked while other threads work on different datasets. This is usually done manually and with great care as it is the heart of both safety and performance. I don't know if there is an automatic solution to this problem that is compatible with the way python programs are structured.

The libraries basically assume that, while you call them, nothing else changes. In order to ensure that you need to lock everything down. Because you don't know what these libraries do and what data they access it needs to be everything (like it is today). It should be possible to only lock the GIL when such a library is called, so there should be kind of a middle way forward.

> Right now you have one mutex for everything (the GIL itself) and everything else doesn't need locking.

If this were true, all of the explicit locking mechanisms in Python's threading module would be pointless. But in fact the GIL's "mutex" is quite a bit more limited than you are saying. It does not prevent all concurrent code from running. It only prevents Python bytecode from running concurrently in more than one thread. But the GIL allows switching between threads in between individual bytecodes, and "one Python bytecode" does not correspond to "one Python statement that performs an operation that you want to be atomic"; plenty of Python expressions and statements are implemented by multiple bytecodes, so it is perfectly possible for multiple threads executing concurrently to modify the same data structures with such statements, creating race conditions if explicit locking mechanisms are not used to prevent it. That's why Python's standard library provides such explicit mechanisms.

> If this were true, all of the explicit locking mechanisms in Python's threading module would be pointless.

Not true. You can have serialized access to the same data structure that still have data race.

But as long as each Python process doesn't keep its local copies for those shared data structures, like free lists, no explicit locking is required if GIL is presented.

> You can have serialized access to the same data structure that still have data race

How?

> as long as each Python process doesn't keep its local copies for those shared data structures, like free lists, no explicit locking is required if GIL is presented.

I have no idea what you're talking about. Different Python processes each have their own GIL, and they don't share data at all (except by explicit mechanisms like communicating through sockets or pipes). Different Python threads share the GIL for their interpreter process, and if each thread doesn't keep its own local copy of data, there is explicit locking required if you don't want data races.

> How?

Simplest scenario, the read-increment-write cycle with 2 threads. Even with a mutex, it is still possible to have data race, if the lock is on per operation level.

For the second part, yep, it is a mistake, not processes, but threads.

With GIL, the thread is given the permission to operate on certain interpreter-related data-structures, like reference counts, or free_list like in PyIntObject. What I mean the active thread is free to modify those data structures without fear of data races, and there is no explicit locking required, if it doesn't hold its own copies of those interpreter internal states.

But GIL can only guard the interpreter's own states, not any user program's states. And yes, explicit locking for operating on your own data is still required.

https://docs.python.org/3/c-api/init.html#thread-state-and-t...

> Simplest scenario, the read-increment-write cycle with 2 threads. Even with a mutex, it is still possible to have data race, if the lock is on per operation level.

What you're describing is not "serialized access with a data race"; it's "multi-thread access that you didn't explicitly control properly".

> For the second part, yep, it is a mistake, not processes, but threads.

Ok, that clarifies things.

> the active thread is free to modify those data structures without fear of data races, and there is no explicit locking required, if it doesn't hold its own copies of those interpreter internal states.

I'm not sure I see why a thread would want to hold copies of those interpreter internal states, since if it did the issue would not be modifying them properly but having the local copies get out of sync with the interpreter's copies, since other threads can also mutate the latter.

The problem with GIL is that it's a mutex that you don't control. So you can't use it to do atomic updates, if that involves more than one low-level operation that is atomic.

So in practice I don't think it simplifies things all that much. If anything, it creates a false sense of security - first developers get used to the fact that they can just assign to variables without synchronization, and then they forget that they still need to synchronize when they need to assign to more than one atomically.

Reply on news.ycombinator.com