Or for a better alternative, just use plain old indices rather than shared pointers.

The scene is only going to be loaded / unloaded all at once, you can just load the data into contiguous arrays and index from them. No need to use shared_ptr since lifetimes aren't that complex.