The assumption was that you'd end up with something close to a microkernel architecture, but without any all-powerful software up top. A purely peer to peer architecture running on bare metal.
I skimmed the paper but I couldn't get a sense of how the CPU scheduler works. What causes the CPU to switch between threads? A common need that I've seen, even in the more limited applications you're describing, is preemption & thread priorities. For example, a driver thread may need to preempt other threads that may be running or we want more of the CPU to be devoted to running some thread even when all threads are runnable. How is this handled?
Re blocking of threads, is the only way they get blocked because they issue an RPC request & thus the CPU then magically knows to switch to a different thread? Is there any other blocking operation that might require needing to switch threads & if so how would that be accomplished?
Another thing I've experienced is that not all HW is DMA'able for cost or space reasons & thus you frequently find such I/O busses to be implemented via bit-banging (e.g. IIRC some times you might have 2 UARTS & only 1 can be used with DMA & the other must be bit-banged). Is that compatible with your approach with all the same guarantees or does this 100% require every I/O system to be part of the DMA network? Or maybe your DMA design is more cost-effective than how CPUs deal with I/O today?
The advantage of a barrel scheduler is that you never can have more than one pair of instructions in the pipeline from the same thread at a given time, so data hazards are impossible and all of the checking/forwarding logic can be entirely absent from the CPU.
You lose single-thread performance with this vs more conventional hyperthreading, but it's much simpler to implement.