See: https://research.nvidia.com/labs/rtr/tag/neural-rendering/

Specifically this one, which seems to tackle what you mentioned: https://research.nvidia.com/labs/rtr/publication/hadadan2023...