However, that leaves a lot of C code that you can't talk to anymore because the C code requires the old Python FFI. I think this is where the main problem lies.
"It's easy to lower the air-conditioning costs of Las Vegas: just move the town to New England".
The problem is "how to remove the GIL" in abstract. It's how to remove the GIL, not impact extensions at all (or as little as possible), keep single threaded performance, and have zero impact to user programs.
To which the above isn't any kind of solution.
Sure, if you don't mind paying a 50-90% performance impact on single threaded performance or completely abandon C-API compatibility and have C extensions start from scratch then there are simple approaches.
If you look at any example in the past to remove the GIL you would see that keeping these two requirements of not having terrible single threaded performance and not having almost a completely new C-API is actually very complex and takes a lot of expertise to implement.
I cannot seem to find much discussion about this. I have found a no-GIL interpreter that works with numpy, scikit, etc. [2][3] so it doesn't seem to be a hard limit. (That said, it was not stated if that particular no-GIL implementation requires specially built versions of C-API libs or if it's a drop-in replacement.)
[0]: https://docs.python.org/3/c-api/stable.html#c-api-stability
[1]: https://docs.python.org/3/c-api/stable.html#platform-conside...
[2]: https://github.com/colesbury/nogil
[3]: https://discuss.python.org/t/pep-703-making-the-global-inter...
Bingo. They don't have to, but often the point of C extensions is performance, which usually means turning on parallelism. E.g. Numpy will release the GIL in order to use machine threads on compute-heavy tasks. I'm not worried about the big 5 (numpy, scipy, pandas, pytorch, and sklearn), they have enough support that they can react to a GILectomy. It's everyone else that touches the GIL but may not have the capacity or ability to update in a timely manner.
I don't think this is something which can be shimmed either or ABI-versioned either. It's deeeep and touches huge swaths of the cpython codebase.
Both C programs can use the GIL for thread safety and can make assumptions about the safety of interacting with a Python object.
Some of those assumptions are not real guarantees from the GIL but in practise are good enough, they would no longer be good enough in a no-GIL world.
> I know that the C-API is only stable between minor releases [0] compiled in the same manner [1], so it's not like the ecosystem is dependent upon it never changing.
There is a limited API tagged as abi3[1] which is unchanging and doesn't require recompiling and any attempt to remove the GIL so far would break that.
> so it's not like the ecosystem is dependent upon it never changing
But the wider C-API does not change much between major versions, it's not like the way you interact with the garbage collector completely changes causing you to rethink how you have to write concurrency. This allows the many projects which use Python's C-API to relatively quickly update to new major versions of Python.
> I have found a no-GIL interpreter that works with numpy, scikit, etc. [2][3] so it doesn't seem to be a hard limit.
The version of nogil Python you are linking is the product of years of work by an expert funded to work full time on this by Meta, the knowledge is sourcing many previous attempts to remove the GIL including the "gilectomy". Also you are linking to the old version based on Python 3.9, there is a new version based on Python 3.12[2]
This strays away from the points I was making, but with this specific attempt to remove the GIL if it is adopted it is unlikely to be switched over in a "big bang", e.g. Python 3.13 followed by Python 4.0 with no backwards compatibility on C extensions. The Python community does not want to repeat the mistakes of the Python 2 to 3 transition.
So far more likely is to try and find a way to have a bridge version that supports both styles of extensions. There is a lot of complexity in this though, including how to mark these in packaging, how to resolve dependencies between packages which do or do not support nogil, etc.
And even this attempt to remove the GIL is likely to make things slower in some applications, both in terms of real-world performance as some benchmarks such as MyPy show a nearly 50% slowdown and there may be even worse edge cases not discovered yet, and in terms of lost development as the Faster CPython project will unlikely be able to land a JIT in 3.13 or 3.14 as they plan right now.
[1]: https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API [2]: https://github.com/colesbury/nogil-3.12
This is exactly the problem, but people have a hard time grasping this because most people interacting with Python have no understanding of how C code interacts with Python, or don't understand the C module ecosystem. I'm not sure if the Python community has a good accounting of this either because I don't recall seeing much quantitative analysis of how many modules would need to be updated etc.
This would help compare with the Python 2 to 3 conversion efforts. Even then, the site listing (shaming?) popular modules with compatibility made a mid-to-late appearance in the process of killing Python 2. Quantification of module updates is obvious thing to have from the get-go for anyone looking to follow through on removing the GIL, but it's not a fun task.
The Thread objects work with old code, but run more slowly.
The GILFreeThread objects are fast.
If an object is passed from a Thread to a GILFreeThread or the other way around, then special safety code is attached to the object so that manipulating the object from the other side doesn't cause issues.
The advantage is that now the module implementers have time to migrate from the old system to the new system. And users can work with both the old modules and "converted" modules in the same system, with minor changes.
But yeah, you will basically have two versions of Python running at the same time, with some (hopefully invisible) translation between them.
Respectfully, I don't believe you have spent any appreciable time looking at the CPython source code. If you had, you would understand how unreasonable this expectation is. I don't say this to tear you down, I say this to convey the magnitude of what you are describing. It would involve touching tens of thousands of LoC. You are talking about a multi-million dollar project that would result in a ton of near-duplication of code.
The red/blue is inescapable because you have to redefine PyObject to have two flavors, PyObject with GIL and GilFreePyObject. You now have to check which one you are dealing with constantly.
No, because if you're running inside a Thread you will know that you will see only PyObjects, whereas if you're running inside a GilFreeThread you will know that you will only see GilFreePyObjects.
If you're manipulating the PyObject (necessarily from a Thread) then there will be behind-the-scenes translation code that will manipulate the corresponding GilFreePyObject for you. But you don't have to know about it.
These are C structs we are talking about here, not some Rust trait you can readily parameterize over abstractly. That either means lots of manual code duplication, or some gnarly preprocessor action. Both are a maintenance nightmare.
I think this is the correct view, because at this moment people are writing various approaches in an attempt at getting rid of the GIL. It's the ecosystem of modules that's the real problem, where you want to basically put in as little effort as possible per module, at least initially.