It seems you are slightly misunderstanding the point of 'unsafe' as a concept.

And no, memory safety is a huge deal, it is just that the borrow checker cannot verify the soundness of certain code, meaning you have to provide the guarantees normally given to you outside 'unsafe' blocks.

Yes, this means that a few data structures require 'unsafe', but you should be creating safe wrappers around these structures; 'unsafe' won't propagate up your code and poison everything.