Trilinear point splatting for real-time radiance field rendering
lfranke.github.io
lfranke.github.io
Currently a huge challenge is in real-time reconstruction. Approaches involve estimating point clouds from images then optimizing splats on those. Other approaches are using SLAM like LiDAR to have distance of points to the camera but then still optimizing on that.
Optimization is producing good results but takes iterations that are not suitable for real-time.
Pixel-wise splat estimation with iPhone LiDAR could produce good results but need help and expertise
https://developer.apple.com/documentation/avfoundation/addit...
Caveat, this field is far outside my wheelhouse. So if not, I'd love to understand more about the nuances here.
Seems like if you added a similar super resolution step to 3DGS you would get similar detail improvements?
On the whole though, this is really interesting and definitely is an improvement over the much older splat techniques.
However for more typical games there's still a long way to go. Most of the research focus has been towards applications where existing mesh-based approaches fall short (eg. photogrammetry), but this isn't really the case for modern game development. The existing rendering approaches and hardware have largely been built FOR games and leveraged elsewhere.
Rebuilding the rendering stack around an entirely new technology is a tall order that will take a long time to pay off. That being said, the technology is promising in a number of ways. You even have games like Dreams (2020), which was built using custom splat rendering to great effect (https://www.youtube.com/watch?v=u9KNtnCZDMI).
I could imagine a hybrid approach of doing the environment with this but other things rasterized as normal and comp'd on top
The big problem though is that the workflow/format is completely different from how current 3D games work so you'd need a lot of custom engine work
I think we'll see some experimental games in the next few years
Think of the difference between vector graphics (like SVG) and bitmap graphics (like JPEG or PNG). While vectors are very useful for many things, it would be quite limiting if they were the only form of 2D computer graphics, and digital photos and videos simply didn't exist. That's where we have been in 3D until now.
1. Have an artist model it by hand. This is obviously expensive. And, will be stylized by the artist, have a quality levels based on artist skill, accidental inaccuracies, etc...
2. Use photogrammetry to convert a collection of photos to 3D meshes and textures. Still a fair chunk of work. Highly accurate. But, quality varies wildly. Meshes and textures tend to be heavyweight yet low-detail. Reflections and shininess in general doesn't work. Glass, mirrors and translucent objects don't work. Only solid, hard surfaces work. Nothing fuzzy.
Splatting is an alternative to photogrammetry that also takes photos as input and produces visually similar, often superior results. Shiny/reflective/fuzzy stuff all works. I've even seen an example with a large lens.
However the representation is different. Instead of a mesh and textures, the scene is represented as fuzzy blobs that may have view-angle-dependent color and transparency. This is actually an old idea, but it was difficult to render quickly until recently.
The big innovation though is to take advantage of the mathematical properties of "fuzzy blobs" defined by equations that are differentiable, such as 3D gaussians. That makes them suitable to be manipulated by many of the same techniques used under the hood in training deep learning AIs. Mainly, back-propagation.
So, the idea of rendering scenes with various kinds of splats has been around for 20+ years. What's new is using back-propagation to fit splats to a collection of photos in order to model a scene automatically. Before recently, splats were largely modeled by artists or by brute force algorithms.
Because this idea fits so well into the current AI research hot topic, a lot of AI researchers are having tons of fun expanding on the idea. New enhancements to the technique are being published daily.
Other examples are things like walking through archeological sites, 3D virtual backgrounds (e.g. for newsrooms), maybe crime scene reconstruction?
It's basically perfect 3D capture, except the big limitations are that you can't change the geometry or lighting. The inability to relight it is probably the most severe restriction.
These research directions are all possible foundations for holographic video codecs, basically. Which is exciting!
Will this (and many others like it) ever be released?
Even provided code is poorly documented and rarely works on a machine other than the author's.
One common phrase is.
"Data is available from the authors upon reasonable request"
Try doing that for any paper older than 18 months.
All it does it contribute to the replication crisis.
As in: you don’t get your name published unless you include a public Git repo URL and a docker container that can run your code.
Otherwise, what’s to stop someone literally photoshopping some made up algorithm presented as vaguely believable pseudocode?
“I have an invisible dragon in my garage, but I can’t open the door right now to show you. Just believe me.”
However in a fast paced field such as this, timing is crucial. And all of these papers so far are technically pre-prints on the arxiv.