Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python | Hacker News Reader

Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python

zacs.site

10 pointszacjszewczyk8 comments

Whenever I see a blog post on Python concurrency that recommends something other than concurrent.futures ThreadPoolExecutor or ProcessPoolExecutor, I shake my head in disappointment at how far we've strayed from "one obviously right way". Why in 2020 am I seeing someone manually close and join a worker pool (blocking!) to return all results all at once because they rewrote executor.map using multiprocessing? Why do they del pool immediately before returning instead of just letting it run out of scope? Why are they using map, which orders the result, if they just want to call all? If performance matters, you want to read the results in the order of completion not the order of submission, and you don't want to wait for all of them to complete if any of them fail. Where did Python's "right way" go sideways?

If I google for python concurrency the second link is to the docs: https://docs.python.org/3/library/concurrency.html

I don’t see where in the docs it says, it’s 2020 so do this from now on. Maybe that’s the problem?

> I don’t see where in the docs it says, it’s 2020 so do this from now on. Maybe that’s the problem?

I guess lack of clear direction is exactly the problem. For all of PEP20's supposed importance, it's notoriously difficult to discover which parts of the standard library to use and which ones to ignore unless you like reinventing wheels.

>First, terms. Most programs work from top to bottom. The next line runs after the last one finishes. We call these linear

Sequential is the term you're looking for.

Linear felt right when I was writing it, for some reason, but you're right. Fixed.

I'm 99% sure threads will run on all cores. I think the author has confused multithreading with hyperthreading.

I was never sure about that, thinking that a single interpreter process runs on one core and the threads must share that process space. But you're right, and I found good confirmation in this visualization (see the "code" link also): http://www.dabeaz.com/GIL/gilvis/fourthread.html

See also http://www.dabeaz.com/python/UnderstandingGIL.pdf, which explains that

    • Python threads are real system threads
    • POSIX threads (pthreads)
    • Windows threads
    • Fully managed by the host operating system
    • Represent threaded execution of the Python interpreter process (written in C)

The article is full of inaccuracies and errors. Confusing threads with eventloops is just one of them.

I can't imagine it's being upvoted.

Reply on news.ycombinator.com