Hacker News Reader

Comment by mkoubaa | Hacker News Reader

Gil free python would break a lot of oss packages written with the C API that are only thread safe because if the GIL.

Couple that with the fact that python on its own without packages isn't that useful and you can see why this is going to be a challenge .

Multiple interpreters in the same process with their own locks is happening now, though

View on HN

Wouldn't it also break most Python code that uses threading? Currently, everything is implicitly synchronized thanks to the GIL, so in Java terms it's as if every variable is declared "volatile".

If the GIL is removed and variables are no longer volatile (i.e, changes are no longer made visible immediately to other threads) that seems like it would break a lot of code. On the other hand, keeping every variable volatile seems like it would be terrible for performance.

Maybe I'm missing some critical difference between Python and the JVM here?

The GILectomy project was able to maintain thread safety on refcounting while (mostly) preserving performance by basically using some clever flags in the refcount to signal specific lifetimes/ownership (simplification for brevity). So you have good performance when the thread owns a reference while preserving safety for shared objects.

The smaller problem is this does at a small amount of overhead even to single-threaded performance. The gilectomy project improved performance elsewhere so net performance is close to the same.

The bigger problem would be integrating this strategy with all the libraries that rely on existing GIL behavior.

>Gil free python would break a lot of oss packages written with the C API that are only thread safe because if the GIL.

So make it a runtime option.

It's getting tiresome that python performance is suffering because some can't be bothered to write thread-safe software. In 2022.

In C which Python extensions use, it hasen't gotten any easier to write thread-safe software "in 2022"

But it has become more necessary and useful, so even C programmers better get used to it (and some tooling to help catch most mistakes).

>so even C programmers better get used to it

Or else? It's not like they're not trying their best - or don't spend the level of effort that they and their companies are willing to take...

I guess it would have made more sense for me to say "Python C extension developers".

I don't find multithreading in languages without particular support easy at all, but I have become better at it. It is possible and sometimes necessary. It seems like the prevailing attitude in the Python ecosystem is weird, a kind of sour grapes thing, i.e. "Python doesn't have good multithreading support, but multithreading is ugly and error-prone anyway and the alternatives are almost as good or better".

Usually when I've needed more parallelization I've allowed more processes and for slow methods, there is threading available (this doesn't overcome the GIL but allows those methods to independently operate). It seems like the biggest reasons to focus on removing the GIL are single-process applications or machines where memory is constrained (so you don't want tons of processes consuming it all). Are you in one of those situations or is there another scenario that is impacted by the GIL?

This comment demonstrates a tremendous amount of naivete about the Cpython runtime. After PyObject itself, the GIL mutex is probably the next most important data structure in the entire codebase. It's not "someone not being bothered to write thread-safe software." It's not something you can hide behind a flag. It's central to the entire cpython data model and any library which relies on releasing the GIL.

The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings, and it's unlikely to ever be upstreamed unless it could be somehow made to work with libraries that rely on assumptions about GIL mechanics (eg numpy).

> The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings

It was Sam Gross and he more or less achieved it:

https://mail.python.org/archives/list/python-dev@python.org/...

Reply on news.ycombinator.com