Comment by geofft | Hacker News Reader

Comment by geofft | Hacker News Reader

From the PEP: https://www.python.org/dev/peps/pep-0554/

> A common misconception is that this PEP also includes a promise that subinterpreters will no longer share the GIL. When that is clarified, the next question is "what is the point?". This is already answered at length in this PEP. Just to be clear, the value lies in:

    * increase exposure of the existing feature, which helps improve
      the code health of the entire CPython runtime
    * expose the (mostly) isolated execution of subinterpreters
    * preparation for per-interpreter GIL
    * encourage experimentation

I think I'll ask the followup question - what is the point of those? Why should we increase exposure of an existing feature we know is not fully baked and we know will cause problems with NumPy/SciPy? How will the exposure improve the code health of CPython and who will do the improvement? What is the advantage in exposing isolated execution of subinterpreters? In what way does exposing this feature help prepare for a per-interpreter GIL? What experiments are being encouraged specifically?

I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale

What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?

The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation.

The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it.

It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)

> multiprocessing allows concurrency at all and this doesn't

I do not think multiprocessing will meet the concurrency needs of many potential applications.

For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me.

For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.

This is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.

I ended up having to switch to dill instead of the default pickle a while back in one of my projects, because pickling lambdas is apparently not possible by default.

But in general it's pretty useful.

Same here.

Everything seemed to be working fine until it didn't.

> To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child.

FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme.

That said,

> I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen.

I have as well, and completely agree.

Came here just to agree.

> Every other major language besides Python allows the former

As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.

Only CRuby has a GIL. JRuby is totally viable for production use and doesn’t have a GIL.

And python also has Jython and other implementations that don't have the GIL. I don't understand what point is being made with that line of thinking?

Does anyone use Jython though? In my experience, only CPython and PyPy are commonly used, both of which have a GIL.

Yes, that's why the work has to start ASAP.

There is currently a project to abstract the C-API with HPy: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e...

The idea is to (very slowly) remove access to direct CPython internals and give a handle to important low-level behavior through a standardized API hidding implementation details.

It's a lot of work, and going to affect a huge part of the ecosystem, but I'm hopeful this is something the extensions authors will actually want to be part of because it promises to solve a lot of problems:

- on the long run, this will make maintaining a c-extension much easier: more stability no matter the evolution of upstream, more certainty.

- there will be a clear way to do things, which means better documentation and conventions, potentially libs to do common things. Right now creating extensions is more a set of receipes people found out to work.

- because of this, scrutiny will be given to common patterns, which will be abstracted and optimized, and given the security of an API contract

- extensions using this API will work out of the box, or faster if they already did, on alternative python implementations like pypy

Of course, the end goal is to free the python core devs to experiment with breaking changes at the c level, such as changing the memory management model, and eventually maybe remove the GIL or allow for a JIT.

My take on this is that subinterpreters by themselves don't excite me that much, but the possibility of having subinterpreters running each in their own thread that do not have to share a single GIL is a huge potential win for Python. No other popular language has the huge restriction on multi-threaded code that Python has with the GIL. I have been hoping for years that the Python dev team would recognize that as an issue and try to do something about it.

> No other popular language has the huge restriction on multi-threaded code that Python has with the GIL.

Is Ruby’s (MRI’s) GVL equivalent to Python’s GIL?

It is.

Reply on news.ycombinator.com