Gaussian splats are much better suited for capturing things where you don’t have artists available and don’t have a ton of performance requirements with regards to frame time.
So things like capturing real estate , or historical venues etc.
It’s definitely not in the cards in the near term without a dramatic breakthrough. Splats have been a thing for decades so I’m not holding my breath.
What am I missing on the performance front?
It also ignores that everything else will be faster too then as well, and ignores needing to target different baselines of hardware.
Either way 5 years for a 3x improvement seems unrealistic. 4 years saw a little over a doubling of performance at the highest end with a significant increase in power requirements as well, where we’re now hitting realistic power limits.
Taking the 2080 vs 4080 as their respective tiers
153% performance increase 50% more power consumption 50% price increase.
So yes performance at the high end will increase, but it’s scaling pretty poorly with cost and power. And the lower end isn’t scaling as linearly.
On the lower/mid end (1060 Ti vs 2060 Super) we saw only a 53% increase in that same time period.
Here's splats from 2020 working at 50-60fps [1]. I think my overall point is I don't think it's performance that's holding it back in games but tooling & whether it saves meaningful costs elsewhere in the game development pipeline.
Otherwise flying cars will also be possible.
Also your splat is running in isolation. Any single system can run by itself at a good clip. That’s not indicative of anything when running as part of a larger system. Again, the discussion of performance is pointless without bounds.
This is not true I believe. There are plenty of papers out there revolving around dynamic/animated splat-based models, some using generative models for that aspect too.
There are also some tools out there that let you touch up/rig splat models. Still not near what you can do with meshes but I think fundamentally it’s not impossible.
With regards to dynamicism, there’s some papers yes but with heavy limitations. Rigging is doable but relighting is still hit and miss, while most complex rigs require a mesh underneath to drive a splats surface. There’s also the issue of making sure the splats are tight to the surface boundary, which is difficult without significant other input.
Other dynamics like animation operate at a very gross level, but you can’t for example do a voronoi fracture for destruction along a surface easily. And again, even at a large scale motion, you still have the issue of splat isolation and fitting to contend with.
The neural motion papers you mention are interesting, but have a significant overhead currently outside of small use cases.
Meshes are much more straightforward, and with advancements in neutral materials and micropolygons (nanite etc) it’s really difficult to make a splat scene that isn’t first represented as a mesh have the quality and performance needed. And if you’re creating splats from a captured real world scene, they need significant cleanup first.
Though I think matterport will just start using them since the other half of their product is the user experience on the web.
They’ll likely never be smaller than a mesh and texture though, because the data frequency will be higher. A wall can be two triangles and a texture. The same representation as splats will have to be many hundreds of points, roughly at the count of the pixels of the lowest resolvable version of that texture.
So I agree they’re far from optimal for data size. But they greatly reduce the complexity of data capture and representation.