It's hard to implement unsafe patterns in safe language. It's hard like using ASM with Java, because Java is platform independent language, while ASM is platform dependent language. This impedance mismatch causes problems, which can solved by wrappers.

If we are talking about emulator, why not just make clear communication between CPU, RAM, and graphic device? Memory can be split into rows (e.g. by split_at()), so graphic device can take ownership of a single scan line at a time, leaving majority of the memory unlocked. We cannot avoid problem, but we can make scale of the problem much smaller by splitting memory into pages, so CPU can take ownership of pages, it works with right now, while graphical device can work with it own pages.