There are devices that do not have access to memory, and you can write a safe description of such a device's registers. The only thing that is inherently unsafe is building DMA descriptors.