Hacker News Reader

Comment by yoyohello13 | Hacker News Reader

It seems like in every python discussion I hear people complain about the GIL.

I’m happy people are working on removing the GIL, but As a professional python dev for about 5 years now I have literally never had a problem where the GIL was a limiter. Although I just make web apps so maybe I’m not the target audience.

View on HN

Conversely, I've worked on backend, data processing-type applications for most of my career, much of it in Java but some (especially recently) in Python, and the GIL is a huge limiting factor for writing efficient, readable Python code. I've had to write very annoying Python code using the multiprocessing library to get around the GIL, and ultimately it works, but it's ugly and clunky and overall just a pain. And remember, I've written a lot of Java code, so I have a high tolerance for pain! But the JVM's concurrency abstractions are actually kind of a joy to use, even if Java the language isn't. Python is the opposite, so if they can shed the GIL and make multithreading viable in Python without forking new processes, that would be a huge win.

I think multiprocessing is quite sensible in Python (comparing to async for an example)

multiprocessing is almost never a good idea in my experience.

It's just about what kind of code you write.

If you write a lot of code that parallelizes over data you will hurt all the time because of the GIL.

If you have worker processes that do something on the CPU, and the results need to be collected and processed further in some other process, you now need to pickle the data to copy it around. That can get really slow.

I understand a lot of python users don't do that kind of thing, but it's a real problem. I'm happy that the python community seems to be slowly beginning to take this seriously after decades of just claiming the GIL isn't a real issue.

> If you write a lot of code that parallelizes over data [and that parallelization can't happen by vectorizing your operations in a C extension module that releases the GIL, like numpy] you will hurt all the time because of the GIL.

I've added some text for clarity

I’ve had the same experience mostly writing web apps in Ruby, which locks similarly to Python. Multi-threading is always needed to saturate IO, network, etc, never CPU, so GIL isn’t an issue. I always wondered what people were doing that ran into issues.

With Python being the language of choice for ML workloads I guess it’s more common to have the CPU be a bottleneck. It seems cool they’re making an option to turn it off for those use cases.

It seems like Python could maintain a GIL compatibility option to preserve the current/old behavior for legacy code.

In my rapidly-approaching-20 years of python development, I have butted heads with the GIL countless times. Are your web apps single-threaded? Have you looked at scaling your service? Or do you use a web framework that handles that for you behind the scenes?

Reply on news.ycombinator.com