Storing the index of an item in a vector in an actual application (vs in some tightly bounded api of a small, purpose-built library used by said application) is highly non-idiomatic and would emit noticeable stink in any code review. It’s just too hard to remember to update all indices after removing an entry. I’ve seen this mistake made in C, C++, rust, C#, JS, and Python code bases and it is usually someone that hasn’t been around long enough to understand the implicit technical debt in certain approaches that would suggest such a thing, regardless of language.
The most obvious/naive solution along the lines you spelled out (in rust but really for any language) would be a hash map of ids/values, otherwise Rc and maybe some weak references.