Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mushed into one value: The light transported. In that sense radiance fields are just 3D photos.
You would have to perform reverse-rendering / photo grammetry and estimate where the light sources and the surfaces are, what materials they have and so on. Then you could use traditional path tracing methods on that again.
Another thing to think about might be videos (not animation): Continuously capture the radiance field over time and then try to compress away the similarities in between frames to gain temporal coherence.
Is depth not properly preserved or something?
Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step.
Here's an edge case I can imagine for dynamic lighting - Say you capture a scene indoors, and a table casts a dark shadow on the floor. But the NeRFs don't try to understand light sources and shadows yet, so it wouldn't know whether the floor is painted black, or a white surface shadowed by the table, or if there's actually a blue Stanford bunny hiding in the shadows.
The 3D scanning rigs that capture small objects like people's faces handle this by manipulating lighting and sampling the BDRF directly. If you can't manipulate the lighting, you can probably guess a BDRF, but there will be limits.
Re-animating might be easy. But capturing an animation, I think you'd need multiple cameras or you'd have to settle for guesswork, like a neural network that can hallucinate the hidden side of a person based on the fact that they're a person. If you point the camera at someone who is walking, you'll get a good view of them from one side, but when you wind the video back, the network won't know what their far side looks like at all.
A few years ago Intel had a project to capture an animated scene with multiple cameras. The pitch was something like, "Just film everything, and you can position the camera in post-processing." I think they wanted it for football games, but I never heard of it shipping. And again, multiple cameras. Matrix-style.
The data requirements might become massive, but there are ways to do the interpolation where it isn’t so bad. If a static scene is 2GB, you should be able to get to a rough time of day approximation in less than 16GB, which is renderable on modern GPUs.
Then it’s “just” a matter of spending several years optimizing it while waiting for H100s to become consumer grade devices.
The only hope is to measure actual photons hitting actual sensors, which is why Gaussian splatting looks so real to begin with.
is a better alternative from your perspective?