Show HN: Real-Time 3D Gaussian Splatting in WebGL

Author here- I'm sorry about the camera controls! Happy to accept pull requests that replace it with something more sensible

The original idea was to be able to navigate around with just arrow keys (conceptually by turning yourself around in place and being able to walk back and forward).

It's very similar to the N64 FPS controls (i.e. Goldeneye): arrow keys (joystick) for the "primary movements" of forward/backward and yaw, with which you can move and look anywhere in a 2D space. Then, WASD (C buttons) for the "secondary movements" of strafe and pitch.

FWIW, OP, I liked the control scheme a lot (using mouse only).

Being brutally honest here, but I just cant get over the control scheme enough to even appreciate the rendering demo. It is unusably unintuitive and awful.

In general, a Gaussian is no longer a true Gaussian after camera projection since the pinhole camera projection function is nonlinear (due to dividing by z). However, if the Gaussian is small relative to the size of the image, you can apporximate it by linearizing the projection function. Therefore the Gaussian splatting paper uses the Jacobian of the projection function as described in equation 5 of the paper [0]. In practice, this approximation is extremely good. This Jacobian is the matrix you mentioned in the third link and it is mathematically sound and not "manually counter balanced". For a derivation, see [1].

[0] https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_...

[1] https://math.stackexchange.com/a/4716514/43771

Yeah I think you're right, they're pretending the projection is a linear transformation (in cartesian coordinates) and using it to transform the Gaussian.

Or viewed alternatively they're approximating the projection by assuming all of the Gaussian is at a fixed depth, which I suppose works if it is far enough away.

A projective transformation of a Gaussian seems somewhat annoying, though I assume someone will have done it before. Seems like it should be possible to do it with projective coordinates but the final projection to cartesian coordinates is tricky.

For what it's worth, projecting a contour is also wrong, the whole density changes which also affects the contours.

Hi. I'm not very familiar with the gaussian splat technique but aren't they essentially quads with some intrinsic data in the vertices. I thought projecting quads was already a solved problem. Could you elaborate how this differs from a simple array of quads? Thank you.

If you can implement the intersecting bounding cone idea without impacting frame rates that's going to be even smoother on WebGPU but it would be interesting to see the difference apples to apples with this type of implementation.

Dynamic? I've. video?

Intentional.

Basically its a semidense point cloud [1], but instead of a point, there is a blob which has been coloured, angled and scaled to match the input picture. This means they are optimised to be viewed from a certain distance.

Think of it like a 3d vector drawing, if you zoom in too much, or pull one part away, it all starts to look a bit funky.

[1]https://www.researchgate.net/publication/326621750/figure/fi...

Depends, radiance field approaches (like gaussian splatting) are basically 3D photos. They do only capture color at geometry (position and direction), but have no concept of surfaces, materials and light transport in general (emission, absorption, transmission, reflection, scattering, etc.). In other words, they can only do static scenes (no animations) with pre-baked lighting.

The industry seems to be trying to move away from this with things like PBR (physical based rendering) and ray / path tracing which enables far better dynamic lighting.

Also, they are extremely space inefficient at the moment. A scene that would take a good traditional rendering engine a few dozen GB would take TB instead. Though, that might improve in the future with more optimization.

One exception to the above, where gaussian splatting might be interesting to see is procedural / generated content (possibly even animated). Especially for volumetric effects which currently use particle systems, like smoke, fire, clouds, flowing water, etc.

Sure, why not? It's just a fancy point cloud. I can easily imagine an open world Minecraft-esque game that uses this for its base engine instead of voxels.

It's already a thing [1]. They also have a project website [2] with some nice videos, although the code hasn't yet been released.

[1] https://arxiv.org/abs/2308.09713

[2] https://dynamic3dgaussians.github.io/

Gaussian splatting is a fancy word for pointcloud but with coloured shapes instead of points.

Its been around for ages, but It was never used because if you have a million points in a point cloud, you'd need to artistically manipulate a million points.

Its like 3d hair, its pretty simple, just render a billion hairs, but in practice its hard to make it look good.

Here we tell a machine learning model to adjust the angle, colour, shape and size of a million primitives (ie a square, circle, triangle etc.) so that it looks like a the photos we provide.

basically this: https://github.com/graphdeco-inria/gaussian-splatting — a somewhat different approach at rendering 3d scenes.

Yes, but this is just the splatting/rendering part and not the optimization part that generates the reconstruction in the first place.

It runs on my mid range phone at 36fps. Did not expect that.

Lots of artefacts though, especially if I move the camera.

Definitely. One of the time-consuming parts of rendering is sorting the gaussians by distance to the camera, which for two nearby cameras could be optimized. This also goes for adjacent frames - assuming smooth motion im pretty sure there is some speedup to be had by assuming the previous sort will be close to correct rather than starting from scratch each frame.

euler angles aren't cool any more