This has been researched quite a lot with the adjacent task of structure from motion (2d videos) - see NeRFs, Gaussian Splatting - and now there are a few models that do 3d from single 2d images fairly well.