Hacker News Reader

Comment by modeless | Hacker News Reader

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's more or less just an optimization. What's interesting is the fact that the knowledge is in there already, learned without 3D input data.

Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!

View on HN

I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)

>The model clearly has an understanding of the 3D structure of objects.

And the submarine clearly can swim! :D

The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.

As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.

https://www.reddit.com/r/Damnthatsinteresting/comments/xrxez...

A machine with 3d understanding from 2d samples predates computers

> Our eyes only ever see 2D images and we learn 3D structure from that

I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.

its not an over simplification. The extra information from convergence is negligible... our eyes derive virtually identical information when looking at flat 2D pictures of 3D scenes. Evidence of this is everywhere in pictures.

I'm sure people born with one eye are perfectly able to understand the concept of 3D

Dunno if you fully read my comment. But we're in agreement.

You're right, we are in agreement.

I think this is a misunderstanding of how these models work. The model does not understand anything at all. It's computing correlations. I could spend all day computing correlation without ever understanding what the correlations correspond to in the physical world. The correlations could still amount to a useful description of some physical phenomenon, but their interpretation requires much more than just the ability to compute correlations in some limited space.

If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.

Your brain merely computes correlations. Does that make it any less intelligent?

At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.

The model is trained on discretized photons (pixels). Doing so may have encouraged the model to learn features related to depth.

Reply on news.ycombinator.com