How to use a Python multiprocessing module
developers.redhat.com
developers.redhat.com
https://replit.com/@Chronological/SlidingPuzzle3 (the code shows multiprocessing Processes connected by queues, the README.md explains what the program does)
Some algorithms aren't easily parallelisable (at least I haven't worked out how to paralellise them!). My original attempts at paralellising A* didn't work because of communication overhead.
But I sharded my data that it searches - each thread searches a different subset of neighbours as A* is a graph search. This lets me paralellise, with 8 threads (when I boosted the repl.it from 2 vcpu to 8vcpu) It would run in 20-40 milliseconds as opposed to 100-300 milliseconds.
mgaunard [0] recently told me that there is something called an algorithmic skeleton for parelllisation problems. I've not used any though. https://en.wikipedia.org/wiki/Algorithmic_skeleton
I have no exposure to high performance computing unfortunately. Just a deep interest in parallelism.
It's path finding between output states of a program. I was inspired by path finding in video games. The path is the discovered program. It's like having a game map where each neighbour of a cell is the potential things you can do at that point in time. For this reason it can be used for unit AI.
I think of assembly (and regular) programming as a sliding image puzzle game. My design is based on this intuitive idea that most programming is just moving things around - it's logistics. How much time do you spend marshalling data? REST APIs? To get registers to the value for a CALL instruction, the calling convention, is just moving the right data into the right locations.
Imagine you have a start state and an end state. It's a turing machine. State is processor register values and memory values in each memory location. There is a series of state transitions between the start state and end state which is created by an instruction, these are intermediary states. Instructions can be MOV instructions or CALL instructions (function calls)
I dynamically generate neighbours of starting from the start state which are potential instructions we can run that get us nearer to the end state. The program searches the space of possible instructions. It infers hidden states caused by functions.
The distance heuristic is the number of values that in registers and memory that match the end state.
My dream: if you knew the type of every API call, you could provide a start state and an end state and the computer would work out the program based on exploring potential instructions that get us to the target state. This is a much simpler form of program synthesis compared to LLMs.
I'll update the README and document the code.
I also have a dream of a real time strategy game where you program the computer by telling unit what to do. In most RTS games you can only build, attack but we can do more.
Also error handling, if one fails I just don't ack the message, after everything is processed I only have failed ones. Can slowly process the messages to see what is wrong
I played about with Kafka consumer groups to get the same effect.
And how do you do that?
Because...that is also what the multiprocessing module does.
It doesn't answer
* why you need multiprocessing (rather than threads)
* what are costs and benefits of them
* how to process a sequence of tasks maintaining results in order
* ...or without maintaining the order
* how to handle exceptions in the processes
* how to gracefully exit by Ctrl+C or other exceptions in the main process
* how to minimize data transmission between processes, because it's costy
etc. None of this is covered.
MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL).
Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs.
Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable).
Exceptions:
Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed.
For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals.
Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.
BTW, I haven't touched MP in Py for couple of years now, but remember there's some misalignment in concurrent futures between pool.map and pool.submit.
And for exceptions in background processes, I had a shared bool var called "emergency brake", and an error queue.
Glad to help, it was similar for me with a lot of effort to synthesize best practices and really appreciate all the concurrent options (asyncio, threads, MP) for where they each are valuable.
If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case).
I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."
My experience, after first getting into python, was:
I needed to do something concurrently on one set of data.
Python threading doesn't provide concurrent execution, so my program slowed down when I used threads.
So, I tried multiprocessing. My program slowed down even more, because any communication between processes uses pickle. I was trying to process one dataset in parallel, and pass big chunks back, for a final processing.
So, I saved it to disk, loaded the dataset into each process, multiplying my memory usage by 16x.
I then threw it all out and wrote the performant bits in C++, using swig to automagically make the python interface for it.
So, knowing why (concurrency) isn't necessarily enough.
Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs:
from multiprocessing import Pool
def f(x):
return x\*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
# to give more of a map reduce flavor:
print(sum(p.map(f, [1, 2, 3])))
Raymond Hettinger, Keynote on Concurrency, PyBay 2017 https://www.youtube.com/watch?v=9zinZmE3Ogk
[0] https://docs.python.org/3/library/concurrent.futures.html#pr...
(This will be fixed in future Python versions, 3.14 maybe.)
from multiprocessing import set_start_method
set_start_method("spawn")It includes a very important observation that lots of problems are small enough that they can be solved with a single CPU core, and that lots of problems are so big you need full on Cloud Computing Distributed Processing with effectively unlimited number of cores, and there are relatively few problems that are too big for 1 core but can be made tractable by going up to around 10 cores. (This is the "Martelli Model of Scalability")
Also this...
Some people, when confronted with a problem, think "I know, I'll use parallelism."
problems
two
they
two
Now
two
two
two
two
two....
The examples are a bit piss weak/simple. One master and one slave. Really I expected 3-4 consumers so you can understand how they interact over q.put() and q.get() and q.join() and q.task_done()
On a large(ish) memory and core host using dnspython I've been getting significantly more dns throughout over log file processing and a dict() cache shared amongst the threads avoiding dns stalls. 20+ threads. It's still buggy, better examples would be helpful.
I know: threads not processes. They're documented as neighbour concepts. I'd have done process forking if I could find good lock and shared mem/mmap use Documented.
https://github.com/tedivm/quasiqueue
I've built a few multiprocess programs with it and it's been really useful, so I thought I'd share.
However, this article feels off to me. Everything I read from redhat always feels a bit wrong for some reason. I don't know what it is exactly. It might just be that they used "a" instead of "the" in the title. But Ive had the same feeling with other manuals and whatnot from redhat.
The most important advise anyone could give on using Python's anything, especially stuff that has to do with concurrency is to stock up on mood enhancing drugs, else they'll probably cry themselves to death out of despair.
I don't understand why a company like RHEL would publish such a low-effort article. But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language... well, never use plain multiprocessing.Process. Use multirpocessing.pool.Pool: https://docs.python.org/3/library/multiprocessing.html#multi...
And if you wonder why would you do that... well, obviously the later builds on the former. And, trust me, it doesn't improve the former in any way. But, it saves you some work. Virtually any task you will end up doing will be done with process pool in less code. Most importantly though, it's a "recognizable pattern" -- it's something that someone who ate from the garbage pile known as Python has learned to recognize in code and to treat accordingly. Code randomly peppered with mention of Process or, worse yet, extending that class is a sign of lack of experience, and produces an urge to delete that nonsense code and write it afresh because nobody's got the time to deal with novice's nonsense.
1000%. Pretty much any other language.
> But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language
Sometimes the drag of dealing with Python multiprocessing is easier and incurs less debt than splitting the project across multiple languages and dealing with (de)serialization of complicated data structures in way that doesn't introduce bottlenecks. And there are at least a few domains where there isn't really a viable alternative to Python.
But a good rule of thumb is that if you're not going to spend at least a couple full days dealing with the issues listed above -- or if you don't absolutely need to use a big chunk of complicated Python code -- you're probably better off picking a different language.
I can somehow only reliably do multiprocessing with numpy when I set:
export OPENBLAS_NUM_THREADS=1
Of course, this is a workaround and not a true solution.
One part of my program uses Numpy, and the other part does multiprocessing. But Python's multiprocessing module causes interference between them (which took quite some time and effort to figure out).