Hacker News Reader

Comment by culebron21 | Hacker News Reader

This covers just the most basic things that can be looked up in the manual (IDK if it's ChatGPT output or not, doesn't matter).

It doesn't answer

* why you need multiprocessing (rather than threads)

* what are costs and benefits of them

* how to process a sequence of tasks maintaining results in order

* ...or without maintaining the order

* how to handle exceptions in the processes

* how to gracefully exit by Ctrl+C or other exceptions in the main process

* how to minimize data transmission between processes, because it's costy

etc. None of this is covered.

View on HN

I don't think I can do all of those justice right this moment but let's get a few here so people can link to an HN thread instead of the red hat article ;)

MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL).

Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs.

Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable).

Exceptions:

Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed.

For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals.

Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.

Thanks. That's more or less how I solved these problems too. And the questions are what I had to painstakingly answer myself at work.

BTW, I haven't touched MP in Py for couple of years now, but remember there's some misalignment in concurrent futures between pool.map and pool.submit.

And for exceptions in background processes, I had a shared bool var called "emergency brake", and an error queue.

I like "emergency brake" as a variable name!

Glad to help, it was similar for me with a lot of effort to synthesize best practices and really appreciate all the concurrent options (asyncio, threads, MP) for where they each are valuable.

Yeah, that's struck me as well. "Let's get down to some work" without explaining why it should (or should not!) be done. And in this case it's crucial.

It's low quality content marketing for Red Hat. Equivalent to all those Medium articles written by boot camp grads, except it's a Red Hat ad instead of a "hire me" ad.

I mean par for the course for tech blogs/articles right.

Agreed, but the bar for HN submissions is slightly higher.

I mean, hopefully why you might need multiprocessing in python is clear?

If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case).

I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."

As a simple example, Threading in Python works good for i/o-bound operations like scraping a website, whereas Multiprocessing works best for CPU-bound operations like result hashing/transforming the data you just scraped.

> I mean, hopefully why you might need multiprocessing in python is clear?

My experience, after first getting into python, was:

I needed to do something concurrently on one set of data.

Python threading doesn't provide concurrent execution, so my program slowed down when I used threads.

So, I tried multiprocessing. My program slowed down even more, because any communication between processes uses pickle. I was trying to process one dataset in parallel, and pass big chunks back, for a final processing.

So, I saved it to disk, loaded the dataset into each process, multiplying my memory usage by 16x.

I then threw it all out and wrote the performant bits in C++, using swig to automagically make the python interface for it.

So, knowing why (concurrency) isn't necessarily enough.

I mean, of course? Concurrency != parallelism, so that makes sense.

Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs:

  from multiprocessing import Pool

  def f(x):
    return x\*x

  if __name__ == '__main__':
    with Pool(5) as p:
      print(p.map(f, [1, 2, 3]))
      
      # to give more of a map reduce flavor:
      print(sum(p.map(f, [1, 2, 3])))

This can be modified to fit a pretty big range of tasks in my use cases and chop hours off my workflows. It's so much faster in DS workflows to just have extra cores and use them than to spin up a cluster for distributed compute.

That's the perfect situation many parallelization packages in python ecosystem take for granted. In my case, I had big datasets that wouldn't fit into memory (or I didn't want to demand this peak RAM usage on a server), and I had to write a parallelizer that read data in chunks and fed it to workers -- something very simple but unavailable in Pandas ecosystem (most packages assume you have swallowed the entire dataset, which isn't that large, and it's easy to throw the data to process-based workers)

I often point people here to get some good background when doing concurrency in Python.

https://realpython.com/python-concurrency/

I've just left the question open, but the best learning material for me on this was this lecture. Other points down the list, I had to figure out just by trial and error.

Raymond Hettinger, Keynote on Concurrency, PyBay 2017 https://www.youtube.com/watch?v=9zinZmE3Ogk

Reply on news.ycombinator.com