What Went Wrong with the Libdispatch

Related; the difference between “red” and “blue” functions is worth a read:

https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...

The freedom to kill and restart the programs as needed makes a huge difference in ease of managing a system. I use kernel-bypass networking and "isolcpus" so the time-critical code never does any system calls. The programs do all their slow, mallocy OS-coupled work at startup, and then never again except to log errors or when shutting down. They mmap structs onto files under /dev/shm/ for status reporting, mostly just incrementing counters that another program, running every few seconds, formats into snapshot reports logged and displayed in grafana.

It is not the right architecture for everything, but it captures to disk all the activity of all the stock and options market exchanges in New York and Chicago using just a single 1U server attached to a 4U disk array.

"Lock-free" operations are not really lock-free at all; they involve atomic bus operations, which just do locking in hardware similarly to what you would otherwise have done. So, to get performance, you minimize interactions enough that the time wasted per interaction is negligible, and batch work so that the branch predictors and caches can get well primed and then used for a stretch. Atomic operations when you are the only writer, and so already own the cache lines, are cheap.

LibDispatch is Apple's re-implementation of SubmitThreadpoolWork WinAPI, introduced in Windows Vista 3 years before libdispatch. Thread pools work better in Windows because OS kernel support.

In libdispatch and similar cross-platform libraries, OS kernel is unaware of the thread pool. The kernel implements individual threads and schedules them fairly, the rest of the code is user mode.

Windows has I/O completion ports (IOCP) in the kernel since forever, more specifically since NT 3.5 shipped in 1994. That newer Vista API is probably built on top, or uses same kernel features.

Here's an important part from IOCP's documentation:

> The system also allows a thread waiting in GetQueuedCompletionStatus to process a completion packet if another running thread associated with the same I/O completion port enters a wait state for other reasons, for example the SuspendThread function.

This is the unique part which can't be implemented in a user mode library. If you create a thread pool with 1 OS thread per hardware thread, and call any blocking API from the pool, a hardware thread will go to sleep. If you create a thread pool with multiple OS threads per hardware thread, the OS will use all of them concurrently and you'll get overhead from context switches, also latency spikes.

The problem can only be solved in OS kernel. That's why in Windows, thread pools usually works OK.

Here's an example of this in Objective-C: https://github.com/stevedekorte/ActorKit

It was a language based on pl/1 with interesting parallel primitives, like cobegin/coend, monitor and a bunch of other things where you could express your parallel needs in different ways. Even years and years later I remember that experience.

Since then I haven't seen cobegin/coend anywhere except in slides, and I haven't seen monitor at all (If I recall correctly it was a critical section allowing only one process inside)

I think the library-based parallel programming scheme is ok, but it might be time to build it into common languages and tools.