It seems we will be having gaussian-splatted 3D world soon.

eg. https://www.nianticspatial.com/products/capture

https://www.youtube.com/watch?v=F-S-x0b9TrE