Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image | Hacker News Reader

Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

worldsheet.github.io

122 pointsArtWomb33 comments

To quote the paper:

> Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time.

Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, but the title seems misleadingly magical.

Very curious to understand this as well. Does that imply that images with similar attributes (like other hotel rooms in one of their examples with slightly different dimensions or furniture) will work without additional training? Or did each example require multiple image training of that scene before a 3d mesh can be computed from arbitrary angle? In the latter case it can still be useful as compared to photogrammetry which would require hundreds of photos to achieve similar results, so maybe this could work with a dozen or less?

Unfortunately that's most(all) of academia!

I don't see how that would work with the paintings.

I think you're misunderstanding the paper. They are using multiple images in the learning phase to infer the 3d structure. At test time, it works on novel scenes with just a single image.

I believe from a second reading this is correct - after training on the Matterport real estate image dataset, it appears they can transform novel images (which happen to be from that dataset, so they are still home interiors but can handle completely distinct scenes with distinct furniture etc.). Which is actually really fascinating and leverages decades of innovation. This is actually really cool and fully justifies the hype in the title.

I'm wondering when we'll get to easy Blade Runner tech, where taking a photo will let you browse it in a 3d sense, extrapolating to some level of certainty what certain aspects "behind" objects or out of view would like look, based off the analysis of the visible elements in the photo.

In lab settings, it seems very close ;)

Detecting Invisible People

http://www.cs.cmu.edu/~tkhurana/invisible.htm

That's not the same thing. This paper is talking about extrapolating objects that become occluded in the video stream until they become visible again, to provide a continuous sense of tracking.

I wonder how long it takes to generate these images. It's not terribly hard to do by hand with rudimentary 3D modeling skills (i.e the most generous way to describe my own 3D modeling capability), if you know the height from the floor and shooting angle at which the camera took the picture. I've done it with 360 photos (therefore shooting angle didn't matter) in Blender with a VR headset to aid in positioning vertices. You can get results like this in about 15 minutes.

Complex scenes obviously take longer, but results can be extremely convincing for a limited range of view angles.

I learned the technique from this video: https://youtu.be/RSC40B8kHT8

15 minutes. I dont know about this particular technique, but others are in the 10-50ms range. Roughly 20k times faster.

My point is, if you need this sort of imagery, you don't need to wait for Facebook to never actually release a usable product out of this paper. Because that's generally what happens with these kinds of papers.

Could you combine many of these images together to make a 3D environment for a computer game, for example inside the bedroom take many photos and then interpolate between the 3D sheets as the player moves around?

That is basically what photogrammetry is all about. It's not easy, but it can work really well when done well.

Didn't read the paper in detail, but seems just an extension / application for single image depth prediction problem?

In the future you could train a model to predict the correct mesh for a given image. Then things get interesting!

That's what they are doing. The rest of the process is not novel.

They're not trying to get the correct mesh, they just treat everything as part of one big continuous surface.

That's good enough for sideways views, but fails if you try to look behind objects, because in their approach, there's no "behind", just a big, stretchy sheet covering everything.

A model with more knowledge about the world would be able to predict that a tree trunk is roughly cylindrical and not connected to the background.

I think this is what they were referring to when they mentioned GANs

This paper does a good job of guessing what should go in a scene when given depths and shapes https://www.youtube.com/watch?v=u4HpryLU-VI&ab_channel=TwoMi...

A combination of the two would be wild

Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.

It's worth making a distinction between "3D Photos where the user supplies depth data from the dual cameras/LIDAR etc" vs "3D Photos where Facebook's magic algorithms try to infer depth". It's the latter that I presume you're criticising here. (Might be obvious to you but I thought it worth spelling out)

I think this paper is also about the latter, it’s just a much better version of the latter - the abstract says RGB image not RGBZ, and one of the examples is a painting.

True, and the latter is indeed what I was referring to. This looks like it does the same thing as Facebook's (train a neural net to infer depth from image contents) but it actually works fairly well.

Generating new views from an rgb+depth image is relatively straightforward and I'd expect any reasonable implementation to work pretty well.

the latter was 6d.ai's whole thing. I say "was" because they were bought by Niantic

Could this be a camera app on the latest iPhones with the built in lidar like technology? You’d snap an image and a mesh, right?

You could, but you don’t need to use a modern phone with a depth sensor – this works with pure RGB and doesn’t need depth information.

It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.

Google Maps Street View already collects some low-res lidar data, so they don't even need this.

They just need to build it into their viewer, which I also wish they would do.

They do use the Lidar data in the viewer. You can see it with the positioning of the cursor on "surfaces" when you're looking around. The reprojection animation they do between photospheres is based on the Lidar data, it's just that's about the best you can do with the low-res data they've collected.

Imagine a situation where you're standing on a street corner with a mailbox or something standing on the corner. You move to the next photo position, which takes you past the mailbox. The smearing and warping you are seeing is from the continous mesh they've created from the Lidar data not matching the real world mesh in areas that were occluded from the original point of view. You get a moment of seeing "behind" the mailbox, but there is no data for what is behind the mailbox, so it all gets interpolated from the data that is known in the surrounding visual area. The mailbox becomes a rectangular prism that extends all the way from the mailbox's location through to the intersection with the ground that we can see behind the mailbox.

These are just the problems with single-image mesh recreation. You can't really get around them without some form of inference of the data that doesn't exist. You even see it in Facebook's images in the linked article, if you look closely. They try to cut the videos early so you don't see it, but it's there if you know what to look for.

> They do use the Lidar data in the viewer.

Oh wow, you're right, I just tried it out. But it's so blocky and apparently limited to (mostly) 90° geometry, that I'd never realized Google was doing anything but modeling a one-size-fits-all "rectangular corridor" along each street.

It seems like it's not exposing any kind of raw Lidar data, but a very simple geometric simplification of it. (E.g. trees are either ignored completely, or if there are enough of them they're treated as the side of a building instead.)

I completely understand what you're saying about the problems with smearing and warping due to not enough data. But I still can't help but wonder what it would look like if they were able to generate a "raw" (but denoised) Lidar geometry, so that trees and cars were treated as individual objects, rather than just either as part of the street floor or part of the building walls they way they are now.

I believe the crude warp is caused by the difficult combination of:

* Need to do animation with very low latency (can't go to the network to collect data for the animation when the user clicks)

* Need to do the animation without much CPU/GPU power in the browser.

* Need to not download much data for the animation ahead of time (browsing panoramas is already very data heavy).

I’ve tried out Apple’s Look Around recently and was impressed with the transition animation compared to Google Maps. If you have an Apple device, try it out in London for instance: Moving past a red telephone box shows how the object kind of retains its shape during the transition. Traffic moving around you also looks quite different compared to Google.

Amazing work, well done!

Reply on news.ycombinator.com