Why would that be the case? I don't think you'd be changing page directory very often for mmio
They're slower, because kernel needs to be mapped in and out of virtual address space, just like for syscalls.
If the access pattern is sufficiently local, perhaps this could be mitigated by using large (2MB) pages. A bad idea for a random access pattern, of course.