Yes! I also started with using vm_memory traits in the beginning but delayed it till I refactor my memory subsystem. When we get that in upstream QEMU it might be possible to share this implementation as well as future devices.
I am curious if you get to DMA before or after QEMU, and if before what your API will look like.