Hacker News Reader

Comment by zerkten | Hacker News Reader

>> However, that leaves a lot of C code that you can't talk to anymore because the C code requires the old Python FFI. I think this is where the main problem lies.

This is exactly the problem, but people have a hard time grasping this because most people interacting with Python have no understanding of how C code interacts with Python, or don't understand the C module ecosystem. I'm not sure if the Python community has a good accounting of this either because I don't recall seeing much quantitative analysis of how many modules would need to be updated etc.

This would help compare with the Python 2 to 3 conversion efforts. Even then, the site listing (shaming?) popular modules with compatibility made a mid-to-late appearance in the process of killing Python 2. Quantification of module updates is obvious thing to have from the get-go for anyone looking to follow through on removing the GIL, but it's not a fun task.

View on HN

This needs more thinking but how about a hybrid approach, where you have Thread objects, and GILFreeThread objects?

The Thread objects work with old code, but run more slowly.

The GILFreeThread objects are fast.

If an object is passed from a Thread to a GILFreeThread or the other way around, then special safety code is attached to the object so that manipulating the object from the other side doesn't cause issues.

The advantage is that now the module implementers have time to migrate from the old system to the new system. And users can work with both the old modules and "converted" modules in the same system, with minor changes.

That sounds like a maintenance and stability nightmare, if it's even possible. You are effectively red/blue splitting the entire codebase. PyObject and the GIL touch everything in the codebase.

The red/blue splitting happens behind the scenes, so it's different. Not really a color problem, because the user doesn't have to know about it.

But yeah, you will basically have two versions of Python running at the same time, with some (hopefully invisible) translation between them.

> But the red/blue splitting happens behind the scenes, so it's different.

Respectfully, I don't believe you have spent any appreciable time looking at the CPython source code. If you had, you would understand how unreasonable this expectation is. I don't say this to tear you down, I say this to convey the magnitude of what you are describing. It would involve touching tens of thousands of LoC. You are talking about a multi-million dollar project that would result in a ton of near-duplication of code.

The red/blue is inescapable because you have to redefine PyObject to have two flavors, PyObject with GIL and GilFreePyObject. You now have to check which one you are dealing with constantly.

> You now have to check which one you are dealing with constantly.

No, because if you're running inside a Thread you will know that you will see only PyObjects, whereas if you're running inside a GilFreeThread you will know that you will only see GilFreePyObjects.

If you're manipulating the PyObject (necessarily from a Thread) then there will be behind-the-scenes translation code that will manipulate the corresponding GilFreePyObject for you. But you don't have to know about it.

What exactly does "running inside a Thread/GilFreeThread" in the context of the cpython runtime mean? You pretty much need an entire copy of the virtual machine code.

These are C structs we are talking about here, not some Rust trait you can readily parameterize over abstractly. That either means lots of manual code duplication, or some gnarly preprocessor action. Both are a maintenance nightmare.

Yes, the assumption is that writing a "double-headed Python" runtime is far less work than converting the entire ecosystem to a new Python runtime.

I think this is the correct view, because at this moment people are writing various approaches in an attempt at getting rid of the GIL. It's the ecosystem of modules that's the real problem, where you want to basically put in as little effort as possible per module, at least initially.

Please read any amount of CPython interpreter code to begin to understand what you’re asking for “behind the scenes”.

This sounds a bit like COM and its apartment-threaded vs. free-threaded objects. The "special safety code" in that case is a proxy object that sends messages to the thread that owns the actual object when its methods are invoked.

Reply on news.ycombinator.com