In short, if you use an unsafe block, then potentially any part of your code is unsafe.
The way I think about it is that the unsafe keyword is a promise that you’re going to maintain the safety invariants yourself, manually. The program is memory correct if it correctly maintains a certain set of invariants. Unsafe punts responsibility over those invariants to the programmer. If you use unsafe and mess up, the program may be in an invalid state. Yes - it might crash, anywhere. But the bug is still almost always in an unsafe block.
It’s often possible to make fully safe wrappers around unsafe code which maintains all those invariants manually. (Either statically or dynamically.). A lot of the rust standard library does this. For example, Vec, Box and slice all use unsafe code internally to create safe APIs.
That's entirely dependent on how you write your Rust code. If you're derefing an invalid pointer then the bug is usually in how you calculated that pointer value, but the only part that actually requires 'unsafe' is the deref, not the bugged pointer calculation.
Now in properly written Rust code you should be marking all of that code as 'unsafe' in that case and documenting what invariants need to be maintained, but that's entirely on you to do. The only part the compiler actually enforces is that you mark the specific spots where you make use of the operations that 'unsafe' allows.
There was a world where everyone just used the unsafe keyword everywhere, and Rust code crashed & had memory issues all the time.
The big thing Rust did wasn't invent borrow checking or memory safety; it's the ease of use, the defaults, and the social aspect of "if unsafe is used, a memory bug is in the unsafe part".
Yes, I wish there was a way to mark code as unsafe without also allowing unsafe operations. Its quite common I have some "safe" code that generates values which are eventually used in an unsafe block. If the "safe" code is wrong, my unsafe code will fail in memory-unsafe ways. But there's currently no way to annotate this sort of "safe" code. Rust provides the unsafe keyword - but that keyword is generally reserved for code which needs to actually make unsafe operations (like derefing pointers or calling other unsafe functions).
The fact that 'unsafe' also enables all unsafe operations in the function body just makes it even messier, since you definitely _don't_ want that unless the entire body is really nothing but unsafe operations. Thus the "mark a safe function as unsafe" has a clear downside since it allows all the unsafe operations you didn't want to use.
This changed in the 2024 edition to now fire a warning lint, asking you to wrap your unsafe operations in an unsafe block, even inside an unsafe function.
This is a common misconception, in a literal interpretation. It doesn't invalidate your point, but since people get the wrong impression: unsafe doesn't turn off any of the checks the Rust compiler does.
It allows you to perform 5 additional operations, and that's it. Using those operations wrong is what breaks the safety promises of the language. As an example, you could dereference a raw pointer and tell the Rust compiler it lives forever, when it's really a pointer to an object that's about to be freed.