Two surprising design decisions:
- One-way messages. You send, then, in a separate operation, you wait for a reply. This happens at each end. That means two extra trips through the scheduler and more time jitter. QNX has a blocking "MsgSend" which sends and waits for a reply. The scheduler transfers control to the receiving thread in the common case where the receiver is waiting, which behaves like a coroutine with bounded latency. It's a subtle point, but one of the reasons QNX is so well behaved about jitter.
- Interprocess communication by memory remapping instead of copying. This is high overhead for small messages, and at some fairly large size, becomes a win. Remapping pages means a lot of MMU and cache churn. Cost varies with the CPU and memory architecture. Mach worked that way, and the overhead was high. Not sure how expensive it is with modern MMUs. Do you have to stop other threads that might have access to the page about to be unmapped?