In principle, the language already has raw pointers with the same expressive power as in C, and unlike references they don't have aliasing restrictions. That is, so long as you only use pointers to access data, this should be fine (in the sense of, it's as safe as doing the same thing in C or Zig).
Note that this last point is not the same as "so long as you don't use references" though! The problem is that aliasing rules apply to variables themselves - e.g. in safe rust taking a mutable reference to, say, local variable and then writing directly to that variable is forbidden, so doing the same with raw pointers is UB. So if you want to be on the safe side, you must never work with variables directly - you must always take a pointer first and then do all reads and writes through it, which guarantees that it can be aliased.
However, this seems something that could be done in an easy mechanical transform. Basically a macro that would treat all & as &raw, and any `let mut x = ...` as something like `let mut x_storage = ...; let x = &raw mut x_storage` and then patch up all references to `x` in scope to `*x`.
The other problem is that stdlib assumes references, but in principle it should be possible to mechanically translate the whole thing as well...
And if you make it into a macro instead of patching the compiler directly, you can still use all the tooling, Cargo, LSP(?) etc.