Deep3D: Automatic 2D-to-3D Video Conversion with CNNs | Hacker News Reader

Deep3D: Automatic 2D-to-3D Video Conversion with CNNs

dmlc.ml

66 pointsstared9 comments

AI-based de-lamination algorithms, which this seems to be, are useful. You'd like to be able to take video frames and break them down into layers which reflect what occludes what. This is not necessarily depth; a single object that recedes into the distance, such as a railroad rail, belongs in one layer. It's relative depth; what's in front of what.

Systems for this should be trained on data with lots of things moving in different directions, occluding each other. Football games, for example. Pedestrians in busy intersections.

This sort of things is a good front end for many image understanding applications. You can also do frameless video compression. Framefree did this a few years ago, but the de-lamination front end of the compressor needed more work. We have more CPU power now; it's time to revisit that.

This is a negative result :) (the method doesn't work)

more like automatic blurry copy of original image :( All the examples are single frames from movies, so why bother when you can extract perfect 3d using multitude of SFM methods (PTAM/SVO/etc) using whole scenes?

I think the authors chose the wrong narrative to discuss in their motivation for the paper. Anyway, it's easier to just shoot a movie / VR content with multiple cameras than go through frame-by-frame to make sure the AI isn't introducing blurry regions etc.

However, being able to extract 3D scene features from a still 2D image is a super useful feature to have. Perhaps in the near future, inference models for image-based tasks (i.e. ImageNet, robotics) will make use of 3D features learned during end-to-end training.

The disparity predictions generated here remind me of spatial transformer layers http://arxiv.org/abs/1506.02025 which Deepmind is using pretty heavily in their sequential attention models. I wonder if they would offer an improvement over the deconv layers.

I think this method managed to tell the foreground from the background. the background is shifted more than the foreground.

you can test the model here: http://www.somatic.io/models/oEG0wMkR the results seem kind of blurry, I hope there is more room to improve this model

Our brains do this very well so there must be some algorithm out there for this.

Well we have two eyes, and each eye provides some depth information (in addition to depth derived from the two perspectives). It's really a completely different problem (Simplified 3d representation of the 3d world).

So you can't tell someone how to navigate a room from a photograph?

Reply on news.ycombinator.com