The way I think about it is that the unsafe keyword is a promise that you’re going to maintain the safety invariants yourself, manually. The program is memory correct if it correctly maintains a certain set of invariants. Unsafe punts responsibility over those invariants to the programmer. If you use unsafe and mess up, the program may be in an invalid state. Yes - it might crash, anywhere. But the bug is still almost always in an unsafe block.
It’s often possible to make fully safe wrappers around unsafe code which maintains all those invariants manually. (Either statically or dynamically.). A lot of the rust standard library does this. For example, Vec, Box and slice all use unsafe code internally to create safe APIs.