As I understand it, unsafe pretty much says "what you are doing here may violate memory safety". Casting doesn't do that, only dereferencing. If you'd like to increase the scope to also include "things that might violate memory safety for another code block", then shouldn't compile either:
let mut foo = unsafe { int_ref as *const u32 as usize };
foo = foo + 1;
let bar = unsafe {*(foo as *const u32)}
the mutation of foo is also "unsafe" under this definition, and the compiler shouldn't let you modify pointers in any manner.