Academic research on this topic is limited, but ongoing. You'd want to look at OSDI and SOSP conferences--they're both OS conferences, even if only a few papers each year are actually about OS design. Lots of ideas there though.

HotOS is a workshop, but tends to be more creative and daring since it's just top researchers sharing their thoughts on OS design.

Off the top of my head in recent years you've had unikernels (e.g., mirageOS), barrel fish (multikernel), omnix (for dealing with lots of asics and other accelerators), and lots of modifications to existing OS design, like a ring-io like asynchronous system call mechanism (flexSC), systems for adding new abstractions off of modern hardware, like Dune, and of course ongoing work on microkernels, most significantly (and commercially), seL4.

Older research is still relevant. Capability systems and exokernels influence the above work and the original well-spring is still worth visiting.

Basically though, what you want to address are some of the following:

* Higher security requirements. Multitenancy is a given. Giant code blobs running without restriction in the same security domain (i.e., monolithic kernels) is a bad idea.

* Greater hardware support for differing security modes (virtualization, enclaves, trust zones, CHERI)

* Advances in compiler/language research and techniques for low-cost policy enforcement at the language level (to at least some degree), allowing for 'uploading' code into ring0 . Rust is the most popular example, but even just the eBPF trend is an example of this.

* Heterogenous hardware. How does the kernel handle differing cores with different performance characteristics, power requirements, capabilities, and even different ISAs? This is happening. Should the OS expose better primitives for ASICS and other accelerators?

* Cloud computing. Many big money work loads run under someone else's control with a hypervisor. This brings up attestation, but also raises the question if you're running a single application under a hypervisor (itself basically a microkernel), what do you need a full fledged kernel for? Similar to userland duplication with containers. Unikernels are one such response, but it's something that should be addressed.

All in all, pretty much anything I see in industry (conceptually) is trialed out in one form or another in academia many years in advance. Industry is theoretically where the really hard work of making it practical happens, but sadly this seems to very rarely happens.