It is trivial to audit the usage of "unsafe". Grep does this. Of course auditing the unsafe functions is another thing. But you can have large codebases without unsafe and consequently a lot of less work with auditing your code.
I don't have personal experience with Rust, but quite a bit with Go. It is almost ridiculous, how much more safe Go is in comparison with C. So yes, it is worth every bit.