Hacker News Reader

Comment by joshuamorton | Hacker News Reader

Python's threadpoolexecutor is still single OS-threaded. There will only ever be one thread executing in parallel (due to the GIL). Basically, in python, asyncio and threadpoolexecutor are almost entirely mappable to each other. Neither can accomplish more than the other.

Multi-process code can, of course, address some of these issues, but it comes at the cost of spinning up multiple OS processes and costly inter-process communication etc.

View on HN

This doesn’t sound quite right. Unless they changed something threadpoolexecutor does involve multiple OS threads, and is not single threaded. Sure the GIL puts a big limitation on parallel execution, but it is most definitely different.

For one, regardless of the GIL it is still possible to have races in multithreaded python code. You have to worry about RMW because the threads are arbitrarily preemptible at a bytecode boundary. This is not the case with asyncio.

Additionally you can release the GIL when making calls to C code. Asyncio is giving you concurrency by using nonblocking calls, you can’t run two CPU bound threads in parallel with it. But you can do this with python threads. So it’s not true that neither can accomplish more than the other.

The only issue I see re: the GIL is that ThreadPoolExecutor (by its name) can mislead them into thinking that they are using proper threads when in fact they are still constrained by the GIL.

It's a confusion that has been quickly resolved every time a team mate of mine has run into the issue, but I have seen it happen multiple times (at different places).

Reply on news.ycombinator.com