I suppose that makes sense when I think about it for a bit. My recent expectations come from work in Rust. There the language prevents you from mutating a map while holding a reference into it. Go doesn't have a mechanism to prevent that (except the one you said, simply not supporting those references at all). If you had a reference into a map that was resized because of a subsequent mutation, your reference would have to keep the whole previous map alive and point to different memory than a reference acquired since then. Both seem undesirable.
With array slots, the same issue is present but is a bit more explicit because those resizes happen with `mySlice = append(mySlice, ...)`.