Tell that to dockerd mounting images layer by layer , with k8s doing all kinds of emptyDir/PVC mounts on top. Pod start up speeds are abysmal now, they would be positively glacial with userspafe permissions validations
Anyways, on linux userland filesystem stuff (ie. fuse) is slow because the kernel manages the vfs layer and has to mediate. In a system designed for non-privileged vfs it doesn't have to be anywhere near so involved. And beyond that, there's really nothing in particular that a filesystem driver does that needs to be privileged to 'run fast'.
VFS is pretty much the posterchild case for how you can pull things out of a kernel and maintain performance, tbh. There's no reason at all it can't be fast except the kernel getting in the way rather than helping.