Super cool to read but can someone eli5 what Gaussian splatting is (and/or radiance fields?) specifically to how the article talks about it finally being "mature enough"? What's changed that this is now possible?
2. Replace each point of the point cloud with a fuzzy ellipsoid, that has a bunch of parameters for its position + size + orientation + view-dependent color (via spherical harmonics up to some low order)
3. If you render these ellipsoids using a differentiable renderer, then you can subtract the resulting image from the ground truth (i.e. your original photos), and calculate the partial derivatives of the error with respect to each of the millions of ellipsoid parameters that you fed into the renderer.
4. Now you can run gradient descent using the differentiable renderer, which makes your fuzzy ellipsoids converge to something closely reproducing the ground truth images (from multiple angles).
5. Since the ellipsoids started at the 3D point cloud's positions, the 3D structure of the scene will likely be preserved during gradient descent, thus the resulting scene will support novel camera angles with plausible-looking results.
Now, perhaps referring to differentiability isn't layperson-accessible, but this is HN after all. I found it to be the perfect degree of simplification personally.
How about this:
Take a lot of pictures of a scene from different angles, do some crazy math, and then you can later pretend to zoom and pan the camera around however you want
- point cloud - fuzzy ellipsoid - view-dependent color - spherical harmonics - low order - differentiable renderer (what makes it differentiable? A renderer creates images, right?) - subtract the resulting image from the ground truth (good to know this means your original photos, but how do you subtract images from images?) - millions of ellipsoid parameters (the explanation previously mentioned 4 parameters by name. Where are the millions coming from?) - gradient descent (I've heard of this in AI, but usually ignore it because I haven't gotten deep enough into it to need to understand what it means) - 3D point cloud's positions (are all point clouds 3d? The point cloud mentioned earlier wasn't. Or was it? Is this the same point cloud?)
In other words, you've explained this at far too high a level for me. Given that the request was for ELI5, I expected an explanation that I could actually follow, without knowing any specific terminology. Do disregard specifics and call it math. Don't just call it math and skip past it entirely: call it math and explain what you're actually doing with the math, rather than trying to explain the math you're doing; same for all the other words. If a technical term is only needed once in a conversation, then don't use it.
Given that I actually do know what photogrammetry is at a basic level, I can make a best-effort translation here, but it's purely from 100% guessing rather than actually understanding:
1. Create a 3d scan of a real-life scene or object. It uses radar (intentionally incorrect term, more familiar) or multiple photographs at different angles to see the 3 dimensional shape.
2. For some reason, break up the stapes into smaller shapes.
This is where my understanding goes to nearly 0:
3-5: somehow, looking at the difference between a rendering of your 3d scene and a picture of the actual scene allows you to correct the errors in the 3d scene to make it more realistic. Using complex math works better and having the computer do it is less effort than manually correcting the models in your 3d scene.
How hard is it to handle cases where the starting positions of ellipsoids in 3D is not correct (being too off). How common is such a scenario with the state of the art? E.g., if having only a stereoscopic image pair, the correspondences are often not accurate.
Thanks.
Is it a fully connected NN?
https://x.com/RadianceFields alt: https://xcancel.com/RadianceFields
For example, the camera orbits around the performers in this music video are difficult to imagine in real space. Even if you could pull it off using robotic motion control arms, it would require that the entire choreography is fixed in place before filming. This video clearly takes advantage of being able to direct whatever camera motion the artist wanted in the 3d virtual space of the final composed scene.
To do this, the representation needs to estimate the radiance field, i.e. the amount and color of light visible at every point in your 3d volume, viewed from every angle. It's not possible to do this at high resolution by breaking that space up into voxels, those scale badly, O(n^3). You could attempt to guess at some mesh geometry and paint textures on to it compatible with the camera views, but that's difficult to automate.
Gaussian splatting estimates these radiance fields by assuming that the radiance is build from millions of fuzzy, colored balls positioned, stretched, and rotated in space. These are the Gaussian splats.
Once you have that representation, constructing a novel camera angle is as simple as positioning and angling your virtual camera and then recording the colors and positions of all the splats that are visible.
It turns out that this approach is pretty amenable to techniques similar to modern deep learning. You basically train the positions/shapes/rotations of the splats via gradient descent. It's mostly been explored in research labs but lately production-oriented tools have been built for popular 3d motion graphics tools like Houdini, making it more available.
https://www.realsenseai.com/products/real-sense-depth-camera...
That said, I don't think splats:voxels as pixels:vector graphics. Maybe a closer analogy would be pixels:vectors is the same as voxels:3d mesh modeling. You might imagine a sophisticated animated character being created and then animated using motion capture techniques.
But notice where these things fall apart, too. SVG shines when it's not just estimating the true form, but literally is it (fonts, simplified graphics made from simple strokes). If you try to estimate a photo using SVG it tends to get messy. Similar problems arise when reconstructing a 3d mesh from real-world data.
I agree that splats are a bit like pixels, though. They're samples of color and light in 3d (2d) space. They represent the source more faithfully when they're more densely sampled.
The difference is that a splat is sampled irregularly, just where it's needed within the scene. That makes it more efficient at representing most useful 3d scenes (i.e., ones where there are a few subjects and objects in mostly empty space). It just uses data where that data has an impact.
It works well for what it does. But, it's mostly only effective for opaque, diffuse, solid surfaces. It can't handle transparency, reflection or "fuzz". Capturing material response is possible, but requires expensive setups.
A scene like this poodle https://superspl.at/view?id=6d4b84d3 or this bee https://superspl.at/view?id=cf6ac78e would be pretty much impossible with photogrammetry and very difficult with manual, traditional, polygon workflows. Those are not videos. Spin them around.
BTW I believe there is software that can turn point clouds into textured meshes reliably; multiple techniques even, depending on what your goals are.
This includes sparse areas like fences, vegetation and the likes, but more importantly any material properties like reflections, specularity, opacity, etc.
Here's a few great examples: https://superspl.at/view?id=cf6ac78e
I would say it's a 3D photo, not a 3D video. But there are already extensions to dynamic scenes with movement.
You generate the point clouds from multiple images of a scene or an object and some machine learning magic
I think this tech has become "production-ready" recently due to a combination of research progress (the seminal paper was published in 2023 https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) and improvements to differentiable programming libraries (e.g. PyTorch) and GPU hardware.
I'm not up on how things have changed recently
tl;dr eli5: Instead of capturing spots of color as they would appear to a camera, they capture spots of color and where they exist in the world. By combining multiple cameras doing this, you can make a 3D works from footage that you can then zoom a virtual camera round.