I don't think this is quite the same comparison. In Rust, multiple mutable pointers to the same object can exist at the same time. So, it's similar to C in this way. It is mutable references that must be exclusive.
Besides the fact I already mentioned about the dangers of casting pointers to references, there's also the problem that pointers are only valid as long as no operations are done with references to the same object (no interleaving). On top of it, the autoborrowing rules make it so it's not always clear when a reference is being taken (and operated upon).
So yes, in my opinion _unsafe_ Rust is significantly more difficult to get right than C.