IMO, using `unsafe` in order to do FFI is of a fundamentally different character than pure-Rust `unsafe`. To the compiler they're the same thing ("I can't verify this, be careful"), but to a reader/auditor the former is fairly reasonable ("I can see why it might be pragmatic to not have to rewrite this entire C program in Rust"), whereas the latter should raise eyebrows ("why, exactly, does this need to be `unsafe`?").
(That said, you obviously still have to be cautious when doing FFI.)