Hacker News Reader

Comment by NotGMan | Hacker News Reader

It's for deep learning, not that much "for graphics".

The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion".

Voxels are bad as they take too much memory and are uniform (each voxel is eg 5x5x5 cm).

Gausians are kinda like "variable sized voxels" that also have orientation, color based on angle viewing and can stretch.

Imagine 3D blobs basicaly (3D capsules or more like 3D density bubbles with transparency).

So the scene can be represented in 3D much more efficiently using gausians splats, which is why they are used for "3d stable diffusion".

So now that this has come into use we also need efficient way or rendering them.

View on HN

Sorry, some objections...

> It's for deep learning, not that much "for graphics".

No, while it is true that there is some overlap between the techniques and concepts used, gaussian splatting isn't necessarily "for deep learning". The library provides a differentiable rasterizer for gaussian splats. This basically means that you can ask it "if I want my output image to change in this and this direction, in what direction should I change the position / orientation / color / ... of my splats?". This enables users to plug it into other software (that is also commonly used for deep learning) and to optimize the parameters of the splats to represent a particular scene.

Since it's primarily a differentiable rasterizer for splats I think it's fair to say that it is "for graphics".

> The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion".

That it uses gradient descent doesn't mean that it is "deep learning". There are no neural networks or layers here.

It's not an "equivalent of stable diffusion". The way it's used now is to learn a representation of a single scene, not unlike photogrammetry. Sure, there may be other use cases for this library, but this is primarily what gaussian splatting is about.

This is not really anything to do with deep learning or AI, except that it uses the same global optimisation algorithm (SGD). But that is a generic optimisation algorithm that can be used for any problem where you can calculate the differential of the loss function.

Would 2d image generation based on gaussian splats be better than stable diffusion?

Nope. Gaussian splatting is for reconstruction of images or 3D shapes not a method for generating them.

Stable diffusion is just reconstructing image from random noise.

What I mean is could you make a version of stable diffusion that uses splats instead of pixels?

the technique really isn’t “stable diffusion” at all. I’ve seen a couple papers build 3D generative models on top of GS though. Also depending on the measure of efficiency, 3DGS isn’t more efficient. Maybe more efficient than other NeRF methods but less than explicitly geometry representation.

And like another person explained, there’s no deep learning. There’s not even a neural net. In the other NeRF literature there are neural nets but they usually aren’t deep. RegNeRF uses a deep neural network along side a shallow net for regularization.

Really nice explanation here.

Reply on news.ycombinator.com