0x02A

Oh I see. That's a pretty cool use case. Not sure how GS would perform on non-photorealistic scenarios, but certainly worth a try.

Does colmap fail on your dataset?

The original paper doesn't work well with few-shot learning. I'm assuming that there is only one camera angle for each pre-rendered background. For single image to 3D, check out DreamGaussian. [1]

[1] https://arxiv.org/pdf/2309.16653.pdf

That's super interesting. Are you trying to do single-view image to 3D?

Yeah, I gave it a try and timings seems to be very wrong. I'll fix that soon.

I haven't tried benchmarking SPIR-V shaders on macOS. Since they're translated into Metal shaders anyways, it should be possible theoretically.

Also, for the command line viewer in the new version, I've only tested make or ninja. I'll take a look at xcode when I get a chance.

Update: I just gave Instruments a try and it seems like the Metal compiler grouped all of the compute and copy operations together and just left the timestamp operations to run back to back. Since MoltenVK isn't a conformant implementation, I'm guessing the synchronization dependencies weren't respected.

However, I'm still getting 200ms frame times on the Garden scene at 4K with M1 Pro. The lego scene shouldn't be too bad even on an Intel Mackbook.

Thanks for trying it out! I haven't had the opportunity to benchmark this on an Intel Macbook. Were you able to see which kernel takes the most time? There should be a performance graph if you have the GUI enabled.

For my Apple Silicon benchmarks, the main bottleneck is the parallel radix sort that sorts the Gaussians by tile and depth. I used a some shaders from a sorting library, but it has some performance gaps with SOTA parallel sort algorithms. I think fixing this would give a 1.5x overall performance boost and maybe 3x on Macbooks. Also the wave size isn't tuned for different GPUs.

Another area of improvement is better management of the shared memory. Right now, we just let the driver manage it as the L1 cache. However, we could manage it manually and group Gaussian retrievals together for the same tile. This is what the official implementation does.

Although 3DGS is the first radiance field with SOTA quality that runs in real-time, I think it's still quite heavy. Due to the explicit representation of the scene, a lot of operations are memory bound. If you can't get an interactive frame rate right now, it's unlikely the improvements will make a material difference.

Hopefully that's where your work on compression comes in and solves the problem :)

Hey HN, this project started out when I was exploring rendering radiance fields in real-time for standalone VR and AR headsets. I was frustrated by the lack of performant, yet non-CUDA, implementations. Also, this would be a good excuse to learn about compute pipelines in Vulkan.

Right now, the renderer runs on Windows, Linux, macOS, iOS, and visionOS (as an iPad app). OpenXR support and an immersive visionOS app are coming soon. Training is also WIP. As we're seeing the industry adopt research in Gaussian Splatting at a fast pace, I hope this makes it easier for folks implementing Gaussian Splatting or variants in their products.

Would love to hear your feedback!

For more context, see previous HN discussions on Gaussian Splatting:

https://news.ycombinator.com/item?id=39120016

https://news.ycombinator.com/item?id=38576974