Hacker News Reader

Comment by etaioinshrdlu | Hacker News Reader

In somewhat related topics, I think we can just use stable diffusion to help convert single photos to 3D NERFs.

1. find the prompt that best generates the image

2. generate a (crude) NERF from your starting image and render views from other angles

3. use stable diffusion with the views from other angles as seed images, refine them using the prompt from 1 combined with(add descriptions to generate "view from back", "view from top", etc

4. feed the refined views back to the NERF generator, keeping the initial photo view constant

5. Generate new views from the NERF, which should now be much more realistic.

Run the above steps 2-5 in a loop indefinitely. Eventually you should end up with a highly accurate, realistic NERF which is full 3d from any angle, all from a single photo.

Similar techniques could be used to extend the scene in all directions.

View on HN

The problem with such an approach would be that NERFs require a set of input images with their exact poses and exact poses are only available if the underlying geometry is static. However if you use SD to generate new views it's only an approximation and you wouldn't be able to get the exact poses.

Not all hope is lost though. I'm pretty sure in a few years (perhaps sooner) we'll be able to generate entire 3D scenes directly without going through 2D images as an intermediate step.

You can optimize the poses as part of the model.

I don't see it as a blocker... Especially if you alternate NeRF and SD iterations - ie. don't generate a whole image each time, instead just choose a random angle, do NeRF reconstruction, then run a single SD iteration on that NeRF, and do another training step of the NeRF.

That way, you know the exact pose for each image, because you chose it when rendering the NeRF.

This 100%. Most images in SD are paintings and have a certain compostion (for instance people usually looking at the camera, posed a certain way, etc). Trying to generate the same thing from a different angle will yield wildly different results which won't work well for Nerf. I do think diffusion models may work for generating 3d stuff, but right now a) We need a big labelled dataset of various 3d objects in different styles (yes, I know some exist but they are nowhere near what we have for 2d images since 2d images can be scraped from the web) 2) Memory/computation is a big bottleneck as 3d is orders of magnitude bigger in terms of memory and processing. That said, hopefully some smart person will come up with some shortcuts that make it possible on today's hardware.

That doesn't work because of the contradiction of NERFs needing multiple images of the same geometry and stable diffusion not producing images with stable underlying geometry because it's purely image-based.

I have my doubts that this will converge to anything meaningful.

In the short term you may be right, but in the long run it's a certainty you won't.

Reply on news.ycombinator.com