PlenOctrees For Real-time Rendering of Neural Radiance Fields (NeRFs)

on your second link I get Error: Unsupported renderer: ANGLE (Intel, Mesa Intel(R) UHD Graphics (CML GT2), OpenGL 4.6 core). Are you running with hardware acceleration enabled?

I much prefer SNeRG's subtitle, Baking Neural Radiance Fields for Real-Time View Synthesis. These techniques are cool but it's very misleading to say they are ‘Real-time Rendering of Neural Radiance Fields’. No, they're real-time renders of data structures sampled from a NeRF.

This matters largely because there is more to NeRFs than just the first paper; eg. there are Mip-NeRF (https://jonbarron.info/mipnerf/) and D-NeRF (https://nerfies.github.io/). A quantized bunch of samples loses the power of neural implicits.

Try 3 days per frame if you want anything in the frame to move/change...

Imagine you have pictures of some real-life object from many angles, and want to turn it into a virtual 3d object (for a web shop, or for use in a computer game, or whatever). Traditional methods try to reconstruct the geometry, but have a couple of problems in more complex situations. NeRFs are a quite new approach that tries to accomplish the same task by training a neural net that essentially allows you to query the color of each point in space (also accounting for the direction you are viewing from). This works great (and works from as few as two source images), but it's slow, and rendering the result was even slower.

This paper introduces a way to render NeRFs at reasonable speed. Still not stellar, but quick enough to make NeRFs useful for many use cases.

Conventional methods of rendering 3D objects and spaces rely on specifying geometry and material properties in some format. You then simulate a viewpoint using that info and physics simulations.

A NeRF takes over both the role of the file format and part of the rendering in the form of a neural network. You feed in a world coordinate and a viewpoint and you get an RGB tuple and density out of it. If you interrogate the NeRF enough you can render any traditional 2D or 3D image out of it by combining all the datapoints.

One theoretical benefit is that that a NeRF is a continuous function, so the resolution is only limited by the capacity of the neural network. Another cool thing is that a NeRF is trained on pictures (with info about where they were taken from), so if you train a NeRF successfully in high-res it’s like scanning an object. A major practical challenge is that it is (was?) pretty frickin’ slow to work with. I wrote a more elaborate comment about it on the previous NeRF improvement post [1]. There I closed with:

> It would be amazing to have NeRF-based graphics engines that can make up spaces out of layers of NeRFs, all probed in real-time.

Here they’ve taken a major step in that direction by speeding up the rendering 3000X.

[1]: https://news.ycombinator.com/item?id=25300283

The linked video shows some information on that https://youtu.be/obrmH1T5mfI?t=288

And in the paper: https://arxiv.org/pdf/2103.14024.pdf

> As training time poses another hurdle for adopting NeRFs in practice (taking 1-2 days to fully converge), we also showed that our PlenOctrees can accelerate effective training time for our NeRF-SH.

Is your camera at equal distance to the object in all photographs? NeRF has troubles with variation in the distance between the object and camera (which Mip-NeRF [0] solves).

[0]: https://youtu.be/EpH175PY1A0

Try https://github.com/AIBluefisher/DAGSfM , it's graph-based approach is much more robust to common repetitive and only-partially-similar image content.

The "Nerf--" paper [1] attempt to tackle that with a directly comparable method. You might also be interested in this [2] slightly different approach that focus on a more explicit decomposition.

[1] https://nerfmm.active.vision/

[2] https://lioryariv.github.io/idr/

I wonder if the days of "digital" content is numbered (from a purely "speculative fiction" point of view)

As someone who consumes plenty of science fiction, I could easily imagine a sci-fi story that breaks information down into some epochs...

epoch 1, "analog": the incoming signal is the raw representation of the data, like grooves on a vinyl record or AM/FM radio

epoch 2, "digital": data is encoded in binary format, and possibly compressed

epoch 3 "???": you don't store the data itself, you store a neural network backed function that can reproduce the data in many different ways based on inputs. I.e. you don't store a video of a chair, you store a neural-net that can approximate the chair from any angle with any light source, PLUS you also store a "camera track" that can be used as a pre-defined input to get a preset "video". But at any time, the user can "unlink" the track and operate the "camera" as they choose.

Automatic, visually accurate 3D visualization of real-world objects from a few images.

Radiance fields are basically photographs that you can move around in (within a limited box). They are especially effective in VR. But, they are huge files. NeRF is effectively AI magic compression for radiance fields.