I do this too, when I can either use a handle/index instead of a pointer, or when I can guarantee that the vectors size is constant (so that pointers/iterators are stable). I’ve also written my own vector that stores its elements in pages so that if its capacity needs to increase, the elements don’t need relocation.
I only really use C++ for a toy game engine right now and in that codebase I don’t use any smart pointers and most objects/functions get passed references to their object dependencies. I classify objects into groups where each groups ownership is very clear. So its owner is responsible for maintaining the memory and any raw pointers can always be assumed to be borrowed references. I use handles then the underlying objects lifetime might differ from whatever is holding a handle to it. Short lived objects are kept trivial and allocated from stack/bump allocators or pools and reset at well defined times (every frame, end of level, etc)
I’m much happier this way than when I used smart pointers or when I had less well defined memory ownership.