SPDK is what folks who really care about this use, I think.
The downside of SPDK is that it is unreasonably painful to use in most contexts. When it was introduced there were few options for doing high-performance storage I/O but a lot has changed since then. I know many people that have tested SPDK in storage engines, myself included, but none that decided the juice was worth the squeeze.
See Ex. https://www.vldb.org/pvldb/vol16/p2090-haas.pdf What Modern NVMe Storage Can Do, And How To Exploit It: High-Performance I/O for High-Performance Storage Engines
for actual data on this.
OFC, If your block size is large enough and/or your design is batching enough etc. that you already don't spend much time in issuing IO/reaping completion then as you say, SPDK will not provide much of a gain.
I believe the next generation ceph OSD is built on seastar as well: https://docs.ceph.com/en/reef/dev/crimson/crimson/
With something like ceph, latency is everything as writes need to be synchronously committed to each OSD replica before the writing client is unblocked. I think for ceph they are trying to move to nvme-of to basically bypass the OS for remote NVME access. Im not sure how this will work security wise however as you cannot just have any node on the network reading and writing random blocks of nvme-of devices.
They use DPDK (optionally) for network IO, not SPDK.
The point of NVMe namespaces is to partition at the block device layer. To turn one physical block device into multiple logical block devices, each with their own queues, LBA space, etc. It's for when your tenants are interacting with the block device directly. That's not a hack, that's intended functionality.