Hacker News Reader

Comment by cft | Hacker News Reader

Then why didn't they remove GIL? Given the end of Moore's law and Python's performance, this alone would have justified the discontinuity.

View on HN

Because the GIL is only a problem in specific sets of problems that the majority of python users won't run into. With the addition of asyncio etc, the negative of GIL is usually pretty restricted to CPU-heavy, blocking services which the core team decided was a very small subset of users. So removing the GIL would require an extremely heavy rewrite of the way python works at a fundamental basis, and it would break a lot of the "promises" that python affords in regards to memory management and safety.

As an aside, there is currently a PEP that will allow for better concurrency __without__ removing the GIL, [PEP554](https://www.python.org/dev/peps/pep-0554/)

and a larger project overall for multi-core support, which is (hopefully) being included in 3.9 [Multi-core](https://github.com/ericsnowcurrently/multi-core-python)

There've been various attempts at this over the years, but it's really orthogonal to changes to the language itself.

Also, check out the SharedMemory objects[1] in 3.8. For my own use cases, this goes a long way toward making me no longer care about the GIL. I'd rather work on multiprocessing code than multithreaded code any time, and if you can easily share objects across processes, then the GIL isn't relevant to anything I'm working on.

[1] https://docs.python.org/3/library/multiprocessing.shared_mem...

I would argue the GIL is a feature put in place to make it easier to write parallel code.

If you want to do this in a more performant way you should be looking at a different language.

People talk about the GIL as if it’s just some drawback that could be removed and everything would be faster. There is a very good reason why it’s there.

> I would argue the GIL is a feature put in place to make it easier to write parallel code.

It doesn't do that though. The GIL protects interpreter internals. It doesn't make Python code thread-safe, although it does make various operations atomic:

* native calls (which doesn't release the GIL) * individual bytecode instructions (which don't call into more python code)

The latter can make it seem like it magically protects your code, but even a simple method call is at least 3 instructions (LOAD_FAST, LOAD_METHOD and CALL_METHOD). Each instruction is atomic and the method call will be if it's a native call (e.g. dict.get) but that's it, the entire call itself is not atomic and you could thread-switch between LOAD_FAST and LOAD_METHOD, or LOAD_METHOD and CALL_METHOD.

You just explained a bunch of implementation details but at the end of the day my point remains: having the GIL makes it easier to write parallel code. I didn't say it magically makes everything threadsafe.

> having the GIL makes it easier to write parallel code.

It does not. What it does is create a bigger trap, because the code looks to be working in more testing situations, and will be more difficult to debug when it starts failing.

Assuming you mean concurrent rather than parallel as the GIL prevents you from running Python code in parallel. How does the GIL make things easier compared to the finer grained locking found in Jython which allows parallel execution of Python code in threads?

Threads may still be switched between instructions every sys.setswitchinterval milliseconds so every instruction boundary is still a potential thread switch point, leading to all the issues described by Glyph in his Unyielding essay: https://glyph.twistedmatrix.com/2014/02/unyielding.html

Isnt the GIL part of the implementation rather than the language?

In the same exact way the garbage collector is. You never touch it but the fact it's there makes it easier to write code.

Because GIL is useful, preventing you from shooting yourself terribly in the foot among other benefits. Languages with better concurrency support (main reason for removing GIL) were designed from scratch for that purpose. You can't remove GIL and keep Python being Python everyone knows. And if you don't mind using another language, there are many choices.

I believe Jython offers the same level of protection against shooting yourself in the foot as CPython, yet it allows allows for parallel execution by relying on finer grained locking. The concurrency issues with threads are still present in CPython with the GIL as the interpreter may switch threads between any instruction.

In addition to the points other people raised, "simply" removing the GIL by changing the current reference-counting GC to do atomic reference updates will also substantially reduce single threaded performance; there was a PyCon talk about that.

https://www.youtube.com/watch?v=P3AyI_u66Bw

See also Swift for another language whose implementation has performance issues caused by ubiquitous atomic reference counting.

Removing GIL will break ecosystem in worse way than Python 2->3 ever did, as both interpreter and compiled libraries rely on it for thread safety.

There are other paths for improving performance that Python is embrancing: asyncio provides considerable performance gains to IO bound apps, and memory-sharing between processes allows for relatively cheap sidestep for those who find threads too slow for their use-case.

Reply on news.ycombinator.com