Hacker News Reader

Comment by VikingCoder | Hacker News Reader

I'm disappointed they didn't demonstrate what happens when they take this system and expose it to a few real-world photos. Can it handle that, or has it been very much fitted to these shapes?

View on HN

"Our method still has many limitations when compared to more traditional computer vision techniques, and has currently only been trained to work on synthetic scenes. However, as new sources of data become available and advances are made in our hardware capabilities, we expect to be able to investigate the application of the GQN framework to higher resolution images of real scenes."

"As new sources of data"...

Like, StreetView?

That could be really interesting, StreetView but with this system imagining interpolated points between the original photos, giving the user more free and natural movement.

More important, more hardware, aka TPUs.

It has very much been fitted. The supplementary material describes the training set as 5 pictures each of 2M different scenes of the type you can see in the paper (square room with random objects).

So to extrapolate wildly, it seems reasonable that getting similar results for, say, real-world bedrooms, you'd need to take around 5 pics each of 2M bedrooms, and record the location and angle of the camera for each picture.

Edit: and I didn't mean to sound negative, using artificially generated rooms to develop the method is a great idea, and the next step will be narrow domain specific applications (they mention eg robotic arms) where it's feasible to automatically collect enough data for a task, and somewhere in the future we may have the data and compute to sample the distribution of real world environments in decent resolution..

FYI, for artificially generated rooms, there is already the SceneNet RGB-D dataset - "5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth"

https://robotvault.bitbucket.io/scenenet-rgbd.html

I think there are only 15k different rooms, rather than 2M you suggested, though!

This could be achieved with a drone camera and positioning sensors.

I'd say much more than that, since real-world bedrooms likely have a much more complicated representation than more simple generated rooms.

I agree with you that the estimate is conservative, and would depend strongly on image resolution and how broad your distribution of bedrooms is - only modern US style, or also 40 year old Japanese houses?

Reply on news.ycombinator.com