Yes, it's safe to cast a reference to a raw pointer and a raw pointer to an address.

It might have been nicer in some ways if the language had restricted that a bit more up front, but it would probably have taken some design work to do that while still supporting a lot of the low-level stuff people need to do.