500 Python Interpreters | Hacker News Reader

500 Python Interpreters

izzys.casa

173 pointsSAHChandler32 comments

So this means modules like concurrent.futures can create a thread pool object (as opposed to processpool) and run in parallel?

Or is the functionality as of 3.13 still limited to low level python embedding applications?

It is expected that many of the more complex Python libraries will break, as they assume no race condition on things that could have now a race condition when GIL is removed.

It will also mean that we can use threads instead of processes and a lot of pain related tot his should disappear in long term.

> "No matter what happens there is going to be a lock occurring. No two PyThreadStates can execute Python bytecode at the same time. However, they can execute multiple C calls at the same time which is why for long running pure C operations extension and embedding developers are encouraged to release the GIL temporarily."

Very exciting! I wonder what the first set of motivating applications are and what kind of performance gains are they expecting.

Lxml frees the GIL internally and iirc the motivation was something like concurrent reads of different portions of large documents.

> // 500 interpreter states

> static constexpr auto MAXIMUM_STATES = 463;

There is a joke here that I’m missing. Does anyone understand what it is?

In the TV show "The Orville" a character has their matter replicator create "500 cigarettes", but it only created 463 after someone counted them all. This is a meme that's been making the rounds for a few weeks in some circles now.

Thanks!

Ah, so that's the origin of that meme TIL, thanks

When they say per interpreter GIL in PEP 684, is that per thread? I wasn't aware of multiple interpreters in one process.

Afaik, you could have multiple CPython interpreters per process before, but the GIL was global to the address space, not a per interpreter instance. I believe the GIL work also put interpreter state behind a single struct instead of a handful of global variables.

This could be totes wrong, all from memory, too tired to fact check.

These were two separate and mostly unrelated PEPs but otherwise that's correct.

Thanks for corroborating, I'll have to trust you.

Putting interpreter state behind a struct allows for multiple CPython instances per address space and each can no run on their own thread independently, they all get their own GI(L) lock. These interpreter instances would still have their own Global to This interpreter Lock, but not a Global lock for all CPython interpreters running in a process (like it was before with interpreter state in globals).

Unrelated but complementary. We could remove the GIL, making Python multithreaded, but if interpreter state was still in a handful globals, you could have only one interpreter per process.

Again, could be totes wrong.

No not per thread. Each interpreter is capable of spawning threads, all of which share that interpreter's gil

Is 463 some joke I'm not getting?

yes: https://old.reddit.com/r/TheOrville/comments/b6yk52/replicat...

Unfortunately, pocketpy moved to global reference to vm recently while changing implementation from C++ to C. Weird decision.

Does this mean that even the Python 3.13 version is around the corner we have to expect some issues when running threads on a program?

Maybe I’m slow today, but are the author complaining (rightfully so) about the buggy no-gil implementation?

Should we wait until the feature is more stable before implementing new libraries? I was thinking on making use of it in new libraries I plan to develop.

What are the most blatant use case to test this feature?

PD: So many questions, I know, sorry everybody!

The article says that embedding (calling Python from C/C++) is currently buggy. It is not clear whether the Python interpreter was compiled with --disable-gil.

I did not check the claim, since, as the article rightly points out, there are general issues in the Python space and other languages like Lua are far better for embedding.

Thanks, I’m not an embedded developer and was losing the train of thought of the author.

These threading issues may also be present when not using embedding. Embedding the interpreter frequently shows general threading bugs much quicker.

In other words, when "torturing" the interpreter with embedding a threading bug might show up after 30 min. The same threading bug might show up once every 6 months in a web server (which is a nasty security risk, since it is hard to detect and to reproduce).

Ouch. So we (app developers) are doomed with this feature?

I did open an issue on GitHub with the CPython implementation and it's being looked into. The crash happens for sure in the debug version of Python. However we're seeing valgrind and address sanitizer fire off for the upcoming 3.13 release in release mode when using the C API as I showed in my example. However, this behavior only appears to occur when embedding python directly, not via extending it, into an application. Because of how the (hidden because it is experimental) `_xxsubinterpreters` module works internally, it's extra internal bookkeeping seems to give enough time for the race condition to disappear.

It might not have been obvious, but I used the currently available Python 3.12.5 release. However upon filing a ticket, and being asked for more testing, we found this issue persists into 3.13 and 3.14 alpha with and without the GIL disabled :(

Thanks for your work and for bringing up this issue to them!

I was hoping for it to be an HTTP status code.

I'm lo0king forward to it, but debugging this stuff is going to be another level of hard.

You posted this 11 days ago and it's been posted by two other people in the interim. (Why is this version of the post more popular?)

Sometimes articles just don't get traction and then they're invited to resubmit (by dang, the moderator). Since the submitter submitted it twice, that was probably what happened here.

As to why it's "more popular", I guess you mean is getting more upvotes, that's part of the trick with submissions. Post it on a slow Friday work day before a holiday and you'll get more clicks than some other days or times.

It was entered into the second chance pool[1]. Interim posts were probably not as popular because I accidentally broke my site's CSS briefly that wasn't obvious in the preview deployments, but was present on the actual site. I only noticed about an hour ago

[1]: https://news.ycombinator.com/item?id=26998308

CSS may still be broken. At least, I'm getting some really obnoxious color combinations (magenta on salmon, etc.)

It should be a brighter orange closer to what you see on HN's top bar. I've been waffling on the magenta. It works "fine" in low usage. This is the first post where I used it a great deal. I'm gonna be spending some time working with the colors over the weekend, as you're the 5th person to say something to me about it.

It could be worse, however. My last design would flashbang you, even if you used darkreader

If you wanna flashback more go for wide gamut colors: https://webkit.org/blog/10042/wide-gamut-color-in-css-with-d...

Reply on news.ycombinator.com