Hacker News Reader

Comment by Znafon | Hacker News Reader

> I would be fine with have it never be removed, and I think we should try sub-interpreter first.

There is a lot of work going on with sub-interpreters and the per-interpreter GIL is shipped in Python 3.12.

The results are very impressive: https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/, as good as one could have hoped I think.

It seems to me like the work on sub-interpreters will continue in parallel ;) to the work on free-threading.

Sub-interpreters and no-GIL have different use-cases though.

View on HN

For now there is nothing impressive with it. The perfs they show off don't include data sharing, which is very primitive.

Which means there is nothing subinterpreters do that can't be done with multiprocessing.

They show progress, and it's good. But I would wait until we can see gunicorn spawning WSGI interpreters workers and get similar performances to the setup with regular workers to get enthusiastic.

to further your point: I've used the technique launching a separate python process for performance critical operations for years on different projects. I can run it in a completely different priority set as well.

Once no-GIL is integrated, what is the use case for sub-interpreters?

I have a use case for sandboxing (without a secure-sandboxing requirement) that it would help solve. But at last read, they discourage/disallow sharing modules across the subinterpreters and reloading all modules in each subinterpreter is not acceptable for my use case.

That's very interesting, thank you for the link. I've used Python since the 00s and I don't ever recall having come across sub-interpreters...

They were always usable using the C-API but not exposed to the Python library (and until 3.12 you were blocked by the shared GIL so there was little point anyway).

Now that the work from Eric Snow has been merged, you can use https://pypi.org/project/interpreters-3-12/ to create one from Python code.

I'd be interested in introducing appropriate abstractions (ABI) and in a module for parallelism. Whereby one can swap out different implementations for sub-interpreters, green threads, GIL or even hardware or cloud vendor-optimised implementations. Not that dissimilar to Project Loom in Java.

Reply on news.ycombinator.com