Making Gaussian Splats more smaller | Hacker News Reader

Making Gaussian Splats more smaller

aras-p.info

91 pointsibobev18 comments

As someone dabbling in the 3D space as a hobby, Gaussian Splats have been the hot trend these past few weeks and everyone in the 3D space is looking into it for getting realistic photo scans into 3D, as an alternative / supplement to NeRF and photogrammetry.

Oh, so that's how Gaussian Splats achieve view-dependence. Strongly reminds me of work on realtime relighting from a decade ago. IIRC some version of Halo used a couple of SH coefficients (either per pixel or per vertex) for some dynamic shadow / lighting thing.

Then, it devolved into figuring out a better set of decomposition functions (SH is Fourier Transform over a spherical coordinate basis). IIRC Haar wavelets worked well too. I'd guess that with modern approaches, a learned function would do even better.

Halo and SH: https://pdfs.semanticscholar.org/566f/c019459435ac3ad25d9a49...

"How does “reflection” of the vase on the metal part of the table work, you might ask? The gaussians have “learned” the ages-old trick of duplicating and mirroring the geometry for reflection!"

Thank you! I'd seen people remove some of the SH coefficients while still getting reflections, and I couldn't understand how that worked. The old tricks sometimes really are the best tricks.

Does make you wonder if you'd see a shadowy vase if you look at the table from below.

Although now that I think about it, there's no real reason to not just make the Gaussians invisible from behind.

I think I've seen so in videos. Like someone dipping a camera underwater and the reflection is real splats.

But since the splats aren't a surface, they don't exactly have a "behind". They might render from every angle on purpose.

> But since the splats aren't a surface, they don't exactly have a "behind".

Spherical Harmonics apply to the whole sphere, splats can learn a SH color for their "behind". But by definition there is no data for these (no camera to tell what the color is). Nothing is preventing current pipelines to define that the opposite direction (splat looking at the camera) has a special color (black, zero-alpha, blend of blurred splats between camera and splat, etc); or to regularize all splats so that there is some amount of an "undefined" transparent component to be applied where camera won't define the SH.

> Nothing is preventing current pipelines to define that the opposite direction (splat looking at the camera) has a special color (black, zero-alpha, blend of blurred splats between camera and splat, etc)

Wouldn't you need to know that there is no other camera view from the opposite direction? (sorry if dumb question, haven't actually looked at how GS are generated from input data)

I can't be considered an expert either, but from what I understand they're currently using spherical harmonics (or some others set of basis functions) to model how a surface emits light from various directions.

However in most cases this will simply cut off as soon as you view a surface from the other side, so it kind of makes sense to add some special handling for that scenario. Especially since it can be hard to properly fit a discontinuity like that.

As it currently stands I imagine the reflection trick would be unable to work if you had a camera view from the other side, which is not ideal.

Oh right, because if we're looking at (say) a ball from two sides, we're not looking at the same splat. So except for very thin 1D-approximating shapes like the spokes of a bike wheel, or pointy convex shapes, most splats will effectively be domes, right?

There's probably some locality you can exploit as well. Given the same surface and close proximity, I imagine the coefficients are nearly identical.

Or store only the rotation of certain typical small set of SH coefficients in two values + index. For an ideal, single material sphere, all the coefficients could be lined up exactly, right?

I think the SH of a splat also depends on geometry around it, for shadows and reflections, so if the sphere is inside a scene, the sphere's SHs may not all be rotations of each other.

Cause I've seen SH used for baked-in shadows.

Very true. But the splats cannot, at the moment, account for moving lights, nor dynamic scenes, so it seems like only the 0-th harmonic would be affected by shadowing.

I looked at that in the previous post (https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-s...) and while there's some nearby correllations in positions & colors (for colors I actually (ab)use that by doing BC7 compression on that data), the rotations & scales are "pretty much random". Which makes sense; if they were not random the GS optimizer would probably have represented them as smaller amount of larger splats.

Feels like the field is testing an changing lots of parameters right now. Hell, we were doing volumetric NeRF not long ago. It's going to be very interesting to where it converges. I would not be surprised to see volumetric wavelets at some point.

Impressive article!

> Of course gaussian splatting at this point is useful for “rotate around a scanned object” use case; it is not useful for “in games” or many other cases. We don’t know how to re-light them, or how to animate them well, etc. etc. Yet.

I know almost for certain that animating them would be a serious challenge, but for relighting, I can more easily imagine some functions that filter some splats based on some kind of spatial and directional querying and applying multipliers for some of their values, however I'm practically daydreaming at this point.

I guess the hard part is reliably figuring out the albedo / color of a darkly shadowed area. It's kind of like dividing by zero.

For materials, you could look at the SH and try to fit it into metallic / plastic / matte.

But they already got photogrammetry into games, and this is just photogrammetry with splats instead of triangles, right? There's got to be a way.

Alternatively just do a Myst style game.

Actually, I think that could be a very nice aesthetic to explore in a hypothetical Myst remake

Reply on news.ycombinator.com