If you need concurrency at the moment, you have already switched to using multiprocessing, so having a no-GIL multithreading is useless.
The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal.
Fixing _that_ should be the objective. What Python need is better support for placing native instances in shared memory.
> The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal.
The PEP goes into the motivation behind this work, and using multiple process does not magically solves all the issues:
> Multiprocessing, with communication via shared memory or UNIX sockets, adds much complexity and in effect rules out interacting with CUDA from different workers, severely restricting the design space.
> I reimplemented parts of HMMER, a standard method for multiple-sequence alignment. I chose this method because it stresses both single-thread performance (scoring) and multi-threaded performance (searching a database of sequences). The GIL became the bottleneck when using only eight threads. This is a method where the current popular implementations rely on 64 or even 128 threads per process. I tried moving to subprocesses but was blocked by the prohibitive IPC costs.
> NumPy does release the GIL in its inner loops (which do the heavy lifting), but that is not nearly enough. NumPy doesn’t offer a solution to utilize all CPU cores of a single machine well, and instead leaves that to Dask and other multiprocessing solutions. Those aren’t very efficient and are also more clumsy to use. That clumsiness comes mainly in the extra abstractions and layers the users need to concern themselves with when using, e.g., dask.array which wraps numpy.ndarray. It also shows up in oversubscription issues that the user must explicitly be aware of and manage via either environment variables or a third package, threadpoolctl. The main reason is that NumPy calls into BLAS for linear algebra - and those calls it has no control over, they do use all cores by default via either pthreads or OpenMP.
and it discusses the alternatives at https://peps.python.org/pep-0703/#alternatives.
You don’t need to share objects. Have explicitly shared buffers instead. Python is a dynamic language so you can easily build proxy objects that are views into a shared buffer, and this allows you to keep all your single threaded performance because no objects are shared.
For example:
buf = sharedbytes.alloc(1 * GiB)
with buf.lock(lockid):
buf[10:20] = 42
message_other_process(buf.id)
# other process
def recv_buffer(bufid):
buf = sharedbytes.get(bufid)
with buf.lock(lockid):
print(buf[10:20])
Wrong. Processes pose hurdles via the limits of IPC and control between processes that Threaded applications don't have to bother with. There are ample examples for this in the PEP.
> What Python need is better support for placing native instances in shared memory.
If that were the case, then threading wouldn't exist in the first place. Shared memory is still IPC. It still requires context switching into the Kernel. It still poses problems that threads don't have.
> It still requires context switching into the Kernel.
There is no less context switching between two threads than between two processes.
> It still poses problems that threads don't have.
Hu, no, it's actually the same problem. If you see threads as different processes with heap being mmap'd at the same location, then you're basically 99.9% right, up to some minor process accounting metadata differences.
I am not talking about switching context between the threads of execution. I am talking about a context switch to kernel code simply to access the shared memory. All interactions with SM require syscalls.
So no, I am not "talking nonsense", and yes, communication between multiple processes requires more context switches than communication between threads.
Please enlighten me with the syscall you use to access a shared memory... because there are none.
As far as the kernel is concerned, "memory" is just a set of pages mapped to a process at a specific address. These pages can be anonymous or named (meaning other processes can map them through that name).
There is no syscall, no context switch, involved to read or write memory, named or anonymous. Hell, that's even the whole purpose of memory mapped IO.
As mentioned earlier, there is also no real difference between a thread and a process as far as the kernel is concerned. A thread is just a special case of process which maps its heap to its parent.
Even the word "thread" doesn't really exist for the Linux kernel. We just call these "lightweight processes", because these are just processes with a shared heap.
Oh rly?
https://man7.org/linux/man-pages/man2/shmat.2.html
https://man7.org/linux/man-pages/man2/shmctl.2.html
https://man7.org/linux/man-pages/man2/shmdt.2.html
https://man7.org/linux/man-pages/man2/shmget.2.html
Pretty sure it says something about "System Calls Manual" at the top of all these man pages ;-)
And IO on the SHM once it's attached may look "free", but it isn't. The M-mapping incurs a further overhead which simply doesn't exist for threads: A threads heap space is the same address space as those of it's siblings in the same process.
1) You linked the man pages of sys5 shared memories. Everyone switched to posix shared memories since literally 20 years.
2) Furthermore, these man pages dont even support your argument. Did you actually read them? These are just for the mapping, not for actually accessing (read/write) the shared memory.
I would suggest, for your own future development, that you care less about trying to look knowledgeable on internet, and more about actually being.
Which is irrelevant, because there is barely any functional difference between the two. POSIX SHM uses a better API, providing a file-descriptor like object. That's all.
And yes, using that API also requires syscalls.
> Furthermore, these man pages dont even support your argument.
Wrong, they absolutely do. "Accessing" something doesn't just involve the IO, it also involves the setup. And this requires syscalls, in SysV as it does in POSIX.
> Did you actually read them? These are just for the mapping, not for actually accessing (read/write) the shared memory
I am well aware of that, hence my seperate mentioning of IO later in my post. ;-)
And the argument in that post stands as solid as it was before. `mmap` MAPS memory. This mapping requires an address translation overhead EVERYTIME THE MEMORY IS ACCESSED. This translation doesn't happen in userspace.
Now, there is a way around that, in principle: If I mmap with ANONYMOUS and SHARED, and then `fork()`, I could have the same mapping in the child process. The problem here is: This relies on the forking after the mmap(). Any newly created objects, if I manage to map them into the child process, will again require address mapping. Again, this isn't an issue in threads. As soon as I create a new object on the heap, it is available to every thread, under the same address.
But hey, what do I know. But I think the people who are going to dedicate countless hours of their lives making actual parallel processing via multithreading possible in Python know why they are doing so. As do the people who implemented abstractions around threading in basically every major programming language ;-)
Well it is relevant, it tells that you're not familiar with the subject and most likely googled what looked related and found outdated man pages which seemed related.
> Wrong, they absolutely do. "Accessing" something doesn't just involve the IO, it also involves the setup
I think common sense would disagree, but if that's your line of arguing, sure.
> And the argument in that post stands as solid as it was before. `mmap` MAPS memory. This mapping requires an address translation overhead EVERYTIME THE MEMORY IS ACCESSED. This translation doesn't happen in userspace.
How deep are you willing to dig your hole there?
Two messages ago you were arguing that accessing a shared memory required syscall and thus was slow. Now you've accepted it doesn't require syscall but argue it requires some magical dynamic translation by the kernel?
I'm sorry but again that is just wrong. All the kernel does is inject the mappings when setting up TLB/MMU at context switch.
The virtual address translation happens without kernel intervention, through the MMU, in _exactly_ the same manner whether this is a named or anonymous memory mapping, whether it is a shared or heap page. In fact the MMU couldn't care less what these addresses are, it has no concept of the meaning of what it translates.
That is an assumption, for which I have yet to see proof :-)
> I think common sense would disagree
Please, do enlighten me: How do I access SHM without mapping it first, or otherwise setting it up?
> Now you've accepted it doesn't require syscall
There was nothing to accept. If you actually read my post, you will see that I talked about the issues of IO and setup seperately.
And my point stands still solid ;-) SHM requires syscalls not required when directly using shared memory.