Comment by usrbinbash | Hacker News Reader

Comment by usrbinbash | Hacker News Reader

Parent Full thread

First of all, these changes are not being introduced because of a committee. They are being introduced because a way to get true thread-based parallelism in Python has been one of THE top priority demands of a huge part of the Python developer community for ages.

> “but now I have to rewrite my library because some people might use it in non-GIL mode”.

Yes, if library maintainers want their library to remain relevant, they will need to accomodate what the languages developer community uses. This is true for all languages. If they don't want to, that's okay, the community will come up with new libraries.

> the committee making these decisions gives zero ducks about the impact this will have

If they were giving zero ducks, they wouldn't make it backwards compatible, nor would there be a command line option to control the behavior.

>Pypi has what, 500k projects on it? Many abandoned. > >Whom exactly is going to update those?

Languages that base decisions on the update behavior, or lack thereof, of library maintainers, effectively freeze themselves.

And why exactly is the update behavior of abandoned packages a problem? They are abandoned anyway.

> Like, sure… it’s a good change for many people… once all the hard work is done by the community.

The people who want to get rid of the GIL are part of the Python development community. Many of them are library developers themselves.

> They are being introduced because a way to get true thread-based parallelism in Python has been one of THE top priority demands of a huge part of the Python

Where is this demand exactly? We hear a lot of complaining but very often this is due to a lack of awareness of available (& often better) alternatives to threading.

There is a very small number of use cases that will benefit from free threading.

> Where is this demand exactly?

The motivation summary of PEP-703 contains some material on this:

https://peps.python.org/pep-0703/#motivation

Further discussions going back years can be found with a brief search. This discussion is almost as old as Python3.

> due to a lack of awareness of available (& often better) alternatives to threading.

Such as?

There are exactly 2: asyncio, which is useless for CPU/GPU bound workloads, and multiprocessing with all the pain of relying on expensive spawns, expensive and limited IPC and the joy of having to orchestrate across process boundaries.

Guess what the most common advice is for dealing with CPU bound parallelisation problems in Python? "Use another language". Guess what all the languages recommended (C, C++, Rust, Go, Java) have in common? They have thread-based parallelism.

> There is a very small number of use cases that will benefit from free threading.

Basically any workload that is CPU bound, which in the day and age of giant data aggragation and running huge ML models at scale is more important than every before, is a use case for this.

> Use another language

Right, because the whole point of Python is as a glue language for native libraries. Adding multi threading to Python is like adding to to Bash.

> Right, because the whole point of Python is as a glue language for native libraries.

I have had this precise need before. I have a multi-threaded native library. The multi-threading is essential for reasonable performance in the intended use-case for the library. But my users can't write C++ to call it; they'd like Python. They'd like to extend certain specific operations that my native library does during processing. I give them Python bindings. The perf gained by running the library on multiple cores is completely negated whenever multiple C++ threads of my native library need to run Python code.

Reply on news.ycombinator.com