I remember having multiple issues doing this in rust, but can't recall the details. Are you sure I would just be able to have whatever refs I want and use them without the borrow checker complaining about things that are actually perfectly safe? I don't remember that being the case.

Edit: reading wavemode comment above "Namely, in Rust it is undefined behavior for multiple mutable references to the same data to exist, ever. And it is also not enough for your program to not create multiple mut - the compiler also has to be able to prove that it can't." that I think was at least one of the problems I had.