This seems like the kind of thing that would benefit a great deal from rust usage.
If not already extant, a FOSS, capabilities-secure, formally-verified RTOS somewhere in the space of seL4 and MINIX 3 microkernel would be useful. Rust might help, but the problem is a lack of attention paid to formal verification all the way downstream, perhaps including symbolic execution and thorough verification of the resulting binaries.
In general, the biggest microkernel design challenges are efficient IPC and handling complex transactions across multiple tasks/processes.
seL4 and such solved IPC performance bottlenecks by proving their moves of message structures were sound under invariants. This is something Rust might be able to solve through moves and lifetimes, but it might not have enough power directly and some edge-cases of unsafe code might be needed for performance reasons.
When it comes to handling privileged events, transactions touching multiple each tasks components must be carefully locked or some privileged helper must step in that violates ordinary constraints. In largely-monolithic Linux, this is relatively easy because when lock-free patterns are unavailable, some or all of the world can be stopped and essentially anything can happen almost anywhere. This simplicity comes at a cost of low/no amounts of isolation that weakens the design goals of a "pure" microkernel architecture which leads to real, hybrid microkernels.
No need for IPC or syscall marshalling when you can simply do function calls without the hassles of MMU permissioning.
Like why is a microkernel needed today for this sort of thing, if its purely for security of memory then... well there's other ways of solving that problem completely outside of hardware memory protection schemes at this point in my mind.
We're so steadfastly stuck in this mindset of needing memory protection and pointer protection (CHERI) because of the insanity that C and its like bring to the table. Restricting the software to safe rust does much the same job. Memory is protected, lifetimes are protected, and better still data races are protected.
E.g. see rtic-rs and how tasks are written. Without unsafe, no memory is accessible outside what is given explicitly to the task. And checking for unsafe rust can be done in an automated fashion at compile time.
Or maybe I'm totally wrong which is quite possible!
Defense in depth is good.
There's a cost to jump from memory protected context (user mode in effect) to supervisor mode, usually an interrupt at minimum. In reality there's typically some syscall mechanism which requires argument marshalling, verification, unmarshalling. Lest you catch a TOCTOU issue.
There's a cost to managing the memory regions... both in wasted memory (page aligned/mpu region aligned) and cpu requirements (reprogramming page tables/mpu regions).
None of that is free of charge and those costs add up quickly.