Doing it with an indexed Vec is basically re-inventing your own memory management system on top of the native one, which as you point out can get very contrived and error-prone. Because then you also have to re-invent allocation/freeing, removal of holes, etc etc.
People sometimes do this in VMs/interpreters where they really do need custom/"unsafe"[1] memory management, which makes sense, but it's definitely not needed for application code like this
[1] Of course it's still memory-safe, but it's more fallible in terms of panics and bugs, as you've pointed out