Hacker News Reader

Comment by milliams | Hacker News Reader

Why does everyone compare against `multiprocessing` when `concurrent.futures` (https://docs.python.org/3/library/concurrent.futures.html) has been a part of the standard library for 11 years. It's a much improved API and the are _almost_ no reasons to use `multiprocessing` any more.

View on HN

Someone downvoted you, I upvoted because I think you have a good point but it would be nice to back it up. I think I agree with you, but I have only used concurrent.futures with threads.

I'll give some more detail. concurrent.futures is designed to be a new consistent API wrapper around the functionality in the multiprocessing and threading libraries. One example of an improvement is the API for the map function. In multiprocessing, it only accepts a single argument for the function you're calling so you have to either do partial application or use starmap. In concurrent.futures, the map function will pass through any number of arguments.

The API was designed to be a standard that could be used by other libraries. Before if you started with thread and then realised you were GIL-limited then switching from the threading module to the multiprocessing module was a complete change. With concurrent.futures, the only thing that needs change is:

  with ThreadPoolExecutor() as executor:
     executor.map(...)

  with ProcessPoolExecutor() as executor:
     executor.map(...)

The API has been adopted by other third-party modules too, so you can do Dask distributed computing with:

  with distributed.Client().get_executor() as executor:
     executor.map(...)

or MPI with

  with MPIPoolExecutor() as executor:
     executor.map(...)

and nothing else need change.

This is why I chose to use it to teach my Parallel Python course (https://milliams.com/courses/parallel_python/).

> Before if you started with thread and then realised you were GIL-limited then switching from the threading module to the multiprocessing module was a complete change

Is this true?

I've been switching back and forth between multiprocessing.Pool and multiprocessing.dummy.Pool for a very long time. Super easy, barely an inconvenience.

I can think of a lot of reasons to use multiprocessing. I do it quite often. You can't always architect things to fit inside of a `with` context manager. Sometimes you need fine grain control over when the process starts, stops, how you handle various signals etc.

The with context manager doesn't seem mandatory. It seems mostly like a convenience to call executor.shutdown() implicitly when the block is done.

I think there is a time and a place for everything. I use concurrent.futures in certain situations when I need to utilize threads/procs to do work in a very rudimentary and naive way. But in more sophisticated systems you want to control startup/shutdown of a process.

TBH, assuming your stack allows it, gevent is my preferred mechanism for concurrency in Python. Followed by asyncio.

For places where I really need to get my hands dirty I will lean on manually controlling processes/threads.

Agreed, concurrent.futures is a great stdlib module.

Too bad it has a somewhat odd name, which doesn't help newbies guess what it really does.

But in almost all cases, it can replace multiprocessing.

i was initially using concurrent.futures for a lot of things and just assumed some of my code wasn't very multiprocessable when i saw it wasn't utilizing all my cores, but it was a speedup nonetheless, then i when i experimented with multiprocessing it gave me considerable speedups with full core usage. i was more happy than frustrated and switched everything i could and got benefit everywhere.

i usually test both when i write code nowadays and concurrent.futures is useful in maybe 10% of my cases.

It's somewhat like metaclasses. As the joke goes: If you have to ask, then you don't need metaclasses.

Reply on news.ycombinator.com