I personally think the Rust community has missed the biggest advantage of unsafe, and that's tooling. Imagine an IDE that could highlight and warn when touching state that is accessed in unsafe blocks, or when state being accessed in unsafe blocks isn't being properly tested.
Or tools that kick off notification for a required code review because code that touches state used by an unsafe block got changed.
To me, these sorts of things will have a far greater effect on security and stability than just the existence of the unsafe block in code.