Last time you used an higher level interpreted language (like JS), did you have to worry whether or not the machine instructions it was generating were the right ones? Generating instructions at runtime sounds pretty unsafe, no?
You didn't, right? You can just mostly assume the language works and get on with your life. That's how unsafe feels to me when I'm programming in Rust, I never have to worry about it and pretty much never need to use it.
Sometimes you'll run into a problem where you need to use unsafe, but that only happens very very rarely, for example when creating an interface to a C library, but most people will never need that. When that happens, sure, you'll have to be extra careful and make sure everything works. But after you're done with that small piece, you're back into safe Rust.
So, how does it work in practice, if you really need to use unsafe? Suppose you have a C enum with values MyEnum {A, B, C} (but it can be any int in C!), and you want to use it in Rust. You can make an unsafe wrapper that tries to parse the enum, and it can either return None - if it was an invalid integer, or return Some(A), Some(B), Some(C).
But once you get the wrapper right (which is not hard), Rust will guarantee that values of type MyEnum have values A, B or C and nothing else, so you never need to worry about the exceptional case again.
You worry about it once at the boundary and that's it.