To be clear, we on the Rust team see 'unsafe' as being strictly about memory safety as well, and have actively fought against using it as a general "pay attention here" kind of annotation.
Eventually, effects would be nice to mark other kinds of "here be dragons" as well, but for now, unsafe == possible memory unsafety.