How about using NERF based techniques? Shouldn't they be much better for this use case?
https://developer.nvidia.com/blog/getting-started-with-nvidi...
There were news about large scale nerfs
Block-NeRF https://waymo.com/research/block-nerf/
CityNeRF https://city-super.github.io/citynerf/
Mega-NeRF https://meganerf.cmusatyalab.org/