DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields
depthoraclenerf.github.io
depthoraclenerf.github.io
But without transparency effects, all of this can be rendered fast and efficiently using geometric textures - i.e. what the Unreal Engine 5 uses. Here's how "traditional" real-time rendering looks like:
https://www.youtube.com/watch?v=PBktSo0bXas
Compared to that, I find "AI rendering" which is blurry and much slower (15fps @ 800px) somewhat underwhelming.
In case anyone here is working on AI rendering, I'd suggest that you focus on transparent glass. That's where all game engines slow down 2x to 4x because of the need to render things in multiple passes. So if you can make fake glass look nice, that's when AI rendering becomes a useful improvement over the state of the art.
As for the scenes in this demo, I'm pretty sure that structure from motion + mesh reduction + normal mapping can reconstruct the original images almost pixel-perfect, and at 200+ fps on modern GPUs. So there's simply not much benefit to having a neural raymarcher over a highly optimized raycaster.
This is actually incorrect, NeRF is the SotA method for these novel view synthesis benchmarks. If you could get better results using meshes, you’d be able to publish a paper and blow all these methods out of the water! That said as you point out there are still major limitations, particularly rendering speed. But the technique is very new and progressing rapidly.
> NeRF is the SotA method for these novel view synthesis benchmarks
That's an interesting statement; uh, I don't know what 'novel view synthesis benchmarks' you're referring to, but the parent post didn't mention them, and like me, probably doesn't care what they are.
If the state of the art is an 800x800 pixel image... uh, well, bluntly, that's really very unimpressive.
> Compared to that, I find "AI rendering" which is blurry and much slower (15fps @ 800px) somewhat underwhelming.
^ This.
It's very much a 'watch this space' technology, because it does have some really interesting and promising features, and it's changing quickly, but I think finding it 'somewhat underwhelming' is a pretty fair response.
I'm flat out skeptical you're not just waving your hands in the air vaguely.
Provide concrete examples of how it's used right now then, specifically in a scenario where traditional techniques don't work.
Not, "look at this youtube video that took 80 hours to render a 800x800 pixel image for our paper"; an actual practical application.
I've never seen NeRF deployed in anything other than a proof of concept or toy scenario.
If you're asking me for an example of it being publicly used in production, well I think you're asking a lot considering the technique is only a few months old.
That is what I explicitly asked for.
You’re failure to provide an example is not because it’s new it’s because it’s actually not useful practically at the moment.
NeRF has been around since March 2020 (https://arxiv.org/abs/2003.08934); you are simply wrong; traditional techniques are better right now, have better implementations and are widely used.
NeRF is a promising technology that is categorically worse in its current implementation and maturity.
I don’t know what else to say.
FYI we once got 1st place on the Sintel AI benachmark in "Clean & EPE matched" with a 2004 paper... By now it's down to 10th place, but AI is by no means far ahead of traditional methods.
As for "novel view synthesis benchmarks", photogrammetry is used in many hollywood productions for virtual actors and/or for virtual environment destruction. In my opinion, having Hollywood use your technique for billion-dollar blockbuster movies is probably a hint that it works well in practice ;)
That said, yes, it's odd they're posing this as a way to extend game engines, when this entire line of research is interesting (to me) for its potential to be a generalized holographic codec. I'm not qualified to evaluate if this is generalizable, but I would hope these results also indicate a possible way forward for camera-derived footage, where there are not acceptable approaches. But to your point:
>As for the scenes in this demo, I'm pretty sure that structure from motion + mesh reduction + normal mapping can reconstruct the original images almost pixel-perfect, and at 200+ fps on modern GPUs.
I don't agree, we've seen attempts to do generalized photogrammetry as a holographic codec, and the results are poor. Microsoft's HoloCapture is probably the best commercial example, and, well... it ain't pretty. (https://holocap.com/) I think the target should be visual fidelity at parity with good 2D codecs. So there's a long ways to go.
The paucity of data sets may be underselling how hard this is and how advanced NeRF (and other approaches like MPI) are relative to advanced photogrammetry. In particular, approaches that extend to video and work well with the human face are very desirable, but there aren't a lot of datasets to test against. And yet these approaches _are_ significant improvements over photogrammetry; I'm not at liberty to share, but I've seen a demo comparing pure photogrammetry to a neural renderer, and the difference is night and day.
But let me agree with you: yes, we should throw these new representations at difficult subject matter, including reflections, translucency, refraction, subsurface scattering, etc, and see how they do. The original NeRF paper had some wild results for refraction, for example, which hadn't been demonstrated before. And when possible, I'd love to see them with video images of people.
I think we're only a few years away from a functional holographic codec, which is pretty exciting.
https://www.russian3dscanner.com/wrap4d/ is popular with Hollywood and it used to cost $10k+ before Epic Games purchased it. And it just happens to be amazing with animated faces ;)
That said, I agree with you in general that NeRF could become amazing if we can get it to work with difficult data sets where photogrammetry breaks down. That's why I find it sad to see them demo it on datasets where photogrammetry is known to work well.