NeRF in the Wild: reconstructing 3D scenes from internet photography

The method is unattended, in the sense that it's photos + camera parameters in and scene representation out. The photos should all be of the same scene (e.g. the Trevi Fountain). Once you have a scene representation, you can ask what the scene would look like from new camera angles with your choice of lighting.

Choosing camera angles is straightforward. You tell me where and what direction the camera is facing. The question then becomes, how do you specify your choice of lighting? The answer is, you can't do so directly. Instead, you provide a picture with the lighting you want, and with a little magic, we can find a way to imitate that lighting. The way we do is by finding a corresponding "appearance embedding" via numerical optimization.

If the question is, "can you reconstruct a (static) scene from the frames in a video?", the answer is yes!

If the question is, "can you reconstruct a scene with people and other moving objects, and model them moving around too?", the answer is not yet.

I do believe that past time travel to reconstructed and recorded events will be one of the stickiest use cases for VR.

Does this work in reconstructing indoor spaces? Give it a shot and find out!

Thank you!

> What was the most challenging aspect of this?

Wow, that's hard to say! Our work truly stands on the shoulders of giants (Mildenhall et al, 2020). I can list off a few challenges: figuring out if an idea "kinda works" or "definitely works" or has a bug, figuring out how to measure progress, coordinating a group of 6 researchers living 9 hours apart, and assembling everything together for a simultaneous paper-website-video release!

> I'm curious to see how you performed edge detection on the transient objects and were able to isolate them so cleanly.

We don't! All of this comes by the magic of machine learning :). We train our model to attentuate the importance of "difficult" pixels that aren't 3D-consistent in the training set. We also partition the scene into "static" and "transient" volumetric radiance fields without explicit supervision. We do so by regularizing the latter to be empty unless necessary, and providing it with access to a learned, image-specific latent embedding. We discard the transient radiance field when rendering these videos, thus removing tourists and other moving objects.

> For some reason, the paper isn't loading, so feel free to say it's explained in detail there.

Well that won't do. Download it here: https://arxiv.org/abs/2008.02268. It's 40 MB, so your download bar may indicate it's almost done, but it actually has a good bit left to go.

> Do you plan on releasing code?

I hope so! As with most code, what ran on our machines may not run on yours. Migrating the code to open source will be a big effort. I hope what we describe in the paper is sufficient to build something like you see here.

The method presented here wouldn't do well with your problem either. 3D reconstruction of moving objects is an unsolved problem!

2) We don't extract surface contours, we learn a volumetric radiance field! To oversimplify, we learn a (smooth) function that, given a position in space, produces the differential opacity and color at that space. To render an image from a camera viewpoint, we approximately integrate along rays emitted from each pixel of the camera.

Check out NeRF and our paper to learn more about this representation!

One of the best non-classical methods is this one (https://grail.cs.washington.edu/projects/sq_rome_g1/), and our method is significantly improves upon it. We do not compare directly with it, but Neural Rerendering in the Wild does, and we improve upon it.

also they're way higher quality than traditional techniques

> and what are z(t) r(t) in equation 5,6?

r(t) is a position in 3D space along a camera ray of the form, r(t) = origin + t * direction.

z(t) is the output of our first MLP. Think of it as a 256-dimensional vector of uninterpretable numbers that represent the input position r(t) in a useful way.

You can use the PhotoSynth tech described above.

I saw in the paper their citation [13] pointed to https://arxiv.org/pdf/2003.01587.pdf, which in section 3 says the following:

We thus build on 25 collections of popular landmarks originally selected in [48,101], each with hundreds to thousands of images.

So hundreds to thousands of photos are used, which is a decent quantity, but definitely makes the quality of the result very impressive.

[Edit] After a little Googling I do see this has been done, using marching cubes (https://www.matthewtancik.com/nerf).