Some kinds of data can be passed back and forth between processes with near zero overhead (no pickling, sockets, or unpickling).
This significantly improves Python's story for taking advantage of multiple cores.
"multiprocessing.shared_memory — Provides shared memory for direct access across processes"
https://docs.python.org/3.9/library/multiprocessing.shared_m...
And it has the example which "demonstrates a practical use of the SharedMemory class with NumPy arrays, accessing the same numpy.ndarray from two distinct Python shells."
Also, SharedMemory
"Creates a new shared memory block or attaches to an existing shared memory block. Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name and a different process can attach to that same shared memory block using that same name.
As a resource for sharing data across processes, shared memory blocks may outlive the original process that created them. When one process no longer needs access to a shared memory block that might still be needed by other processes, the close() method should be called. When a shared memory block is no longer needed by any process, the unlink() method should be called to ensure proper cleanup."
Really nice.
With mmap you have to specify a file name (actually a file number), but so long as you set the length to zero before you close it there's no reason any data would get written to disk. On Unix you can even unlink the file before you start writing it if you wish, or create it with the tempfile module and never give it a file name at all (although this makes it harder to open in other processes as they can't then just mmap by file name). The mmap object satisfies the buffer protocol so you can create numpy arrays that directly reference the bytes in it. The memory-mapped data can be shared between processes regardless of whether they use the multiprocessing module or even whether they're all written in Python.
I thought that when you use multiprocessing in Python, a new process gets forked, and while each new process has separate virtual memory, that virtual memory points to the same physical location until the process tries to write to it (i.e. copy-on-write)?
Empty space in internal pages gets used allocating new objects, refence counts updated or GC flags get flipped etc, and it just takes one write in each 4kb page to trigger a whole page copy.
It doesn't take long before a busy web worker etc will cause a huge chunk of the memory to be copied into the child.
There are definitely ways to make it much more effective like this work by Instagram that went into Python 3.7: https://instagram-engineering.com/copy-on-write-friendly-pyt...
Sharing post-fork data is where it gets interesting.
E.G: live settings, cached values, white/black lists, etc
But still copying?
If not, then how does it interoperate with garbage collection?
So it's not for containing normal Python dicts, strings etc that are individually tracked by GC.
https://docs.python.org/3.8/library/multiprocessing.shared_m...
Would this work with e.g. large NumPy arrays?
(and this is Raymond Hettinger himself, wow)