Python's GIL implemented in pure Python | Hacker News Reader

Python's GIL implemented in pure Python

rushter.com

127 pointsetrevino18 comments

Reading the headline I couldn’t help but think “But why?”. Actually quite interesting and valuable implementation info if you’re ever doing threading with CPython.

> Reading the headline I couldn’t help but think “But why?”.

Reading the article will clarify.

Reading the rest of the post you're replying to might help.

Yes, I think the headline would be better as "Python's GIL explained in Python code"

When I first read about pypy a decade ago, I also asked myself "but why". Now it is one of the most interesting projects out there and any talk they give tends to be really worth listening to. Their talk on Software Transactional Memory blew my mind.

This is nice .. but it could be nicer with (pun intended) use of the context manager protocols, a few less empty lines and a bit of minor refactoring for readability.

Is execution_loop really meant to start and end holding the GIL?

Two empty lines before function definitions is part of the PEP8 style standard, which is widely used in the Python community. Most Python autoformatters and linters default to PEP8.

As for empty lines within code, I personally find it helpful to visually separate logical segments.

> As for empty lines within code, I personally find it helpful to visually separate logical segments.

I see it as, if a function is like a paragraph, then the whitespace between lines deliminates sentences. I very much dislike run on sentences that are attempting to explain complex rules/topics in english and in code. Spatial grouping is very natural for the eyes and mind.

> Note that this code will not run if you will try to execute it, because it's missing bytecode execution logic.

So much for "implemented".

Incidentally, apparently PyPy has a GIL too[1], complete with a detailed description in a source-code comment, similar to the one quoted in the linked article[2]. It's written in C. Isn't the point of PyPy that it's written in a Python subset?

[1]http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-... [2]https://bitbucket.org/pypy/pypy/src/00f8eae1/rpython/transla...

> Isn't the point of PyPy that it's written in a Python subset?

That subset is rpython (which beyond a python subset is a framework for implementing dynamic language runtimes: https://rpython.readthedocs.io/en/latest/).

You'll note that the gil is part of rpython.

> So much for "implemented".

This comment reeks of middlebrow dismissal. The author implemented the GIL in pure Python for illustration purposes, as it says in the very first paragraph of the article, yet you complain it won't actually run. Where were you thinking of using a pure-Python GIL that this is such an inconvenience to you?

> Isn't the point of PyPy that it's written in a Python subset?

More dismissal. The point of PyPy is to be a JIT. What it's written in is incidental, not "the point".

My first point was a bit of a nitpick, sure, but I'd never describe my work as 'implementing' something unless it actually worked, in full.

As to my second point, you're reading dismissal where there is none intended. My question wasn't rhetorical. Ironically, you've just dismissed my question.

Is Wikipedia's PyPy article mistaken in describing it as 'Written in RPython'? I'll go ahead and fix it if so, or perhaps I'm missing something about the status of the C code.

What word would you use to describe writing something in a language for illustration purposes?

PyPy is written in RPython. You posted a link to the RPython translator's source code.

> What word would you use to describe writing something in a language for illustration purposes?

I'd probably go with "The GIL explained in Python".

> PyPy is written in RPython. You posted a link to the RPython translator's source code.

But RPython is a part of the PyPy project, no?

Do you consider the Wikipedia listing to be accurate?

RPython is part of the PyPy project, yes. PyPy the interpreter is written in RPython. Parts of the RPython translator (maybe all of it, I'm not sure) are written in C. RPython is part of the PyPy project, but is not part of the interpreter (it just compiles it).

I do consider the Wikipedia listing accurate, yes.

"PyPy" can mean two things: either the PyPy interpreter, or the PyPy project. The PyPy interpreter is written in Python, you could run Python on PyPy running on CPython if you wanted. (The PyPy folks probably do that regularly to debug the interpreter.)

Because PyPy on CPython is never going to be faster than CPython alone, the RPython toolchain is used to convert the PyPy interpreter to C code. The C code in the RPython source (e.g. the GIL implementation) is bundled together with the translated program and then compiled into machine code.

RPython is also written in Python. It works by importing the program to be translated, executing all initialization code to create classes, methods and global objects. This is no different from regular program startup. But then RPython is provided with a particular function as an entry point, and it does whole-program type inference to determine a static type for every reachable value. That only works because PyPy is written in a style that only uses dynamic Python features during initialization and is very restricted after the entry point. So PyPy is actually written in the subset of Python that RPython accepts.

If you tried to use RPython on any arbitrary Python program, it'd probably complain about a ton of type errors. RPython is really a very different language, and its feature set is tailored specifically to writing PyPy in it. It can handle certain constructs that Mypy isn't able to typecheck because they require running the program, but it also rejects many well-typed Python constructs because they don't translate to efficient C.

> My first point was a bit of a nitpick, sure, but I'd never describe my work as 'implementing' something unless it actually worked, in full.

The GIL is fully implemented. Just because it's typically seen with a bytecode interpreter doesn't make that such an interpreter is part of it, or a prerequisite.

This code describes all limitations of the multithreading in Python and kinda explains why Python can't use multiple cores despite using real threads.

Why would you need a working GIL inside the GIL?

Reply on news.ycombinator.com