That's entirely dependent on how you write your Rust code. If you're derefing an invalid pointer then the bug is usually in how you calculated that pointer value, but the only part that actually requires 'unsafe' is the deref, not the bugged pointer calculation.
Now in properly written Rust code you should be marking all of that code as 'unsafe' in that case and documenting what invariants need to be maintained, but that's entirely on you to do. The only part the compiler actually enforces is that you mark the specific spots where you make use of the operations that 'unsafe' allows.