3-D Depth Reconstruction from a Single Still Image (2007) [pdf] | Hacker News Reader

3-D Depth Reconstruction from a Single Still Image (2007) [pdf]

cs.cornell.edu

96 pointsJasamba28 comments

This is some very cool research, though many may be surprised to learn that it is an IJCV journal article from 2007, based on a conference paper from NIPS 2005.

Source: http://www.cs.cornell.edu/%7Easaxena/learningdepth/

Gah! I thought this was cutting edge. Is this research used in any recent works/research?

Definitely. It's been cited 658 times since it was published:

https://scholar.google.com/scholar?oi=bibs&hl=en&cites=18064...

As an aside, I really enjoy it when people post articles like this on HN. Technical, but not too niche, and relevant to today's interests in technology.

Video is a nice watch, especially as this is from 2007, so the state of the art should be even better. https://www.youtube.com/watch?v=UZ7_ED9g4FY

Google Tango (https://www.google.com/atap/project-tango/) is able to do pretty impressive 3d reconstruction from what I hear. However it has a number of cameras, whilst linked technique can work with just one image (with enough training).

reconstructing depth from 2+ offset images is vastly simpler than from a single frame.

There's an OpenCV tutorial for those wishing to try it[0].

[0] http://docs.opencv.org/3.0-beta/doc/py_tutorials/py_calib3d/...

Yup, but I think there's a desire to get good 3d-reconstruction techniques for single images because in practice you often want to do reconstruction in extremely small environments where using multiple cameras is sometimes not feasible for various reasons.

Google Camera can already do this from a single camera. It takes a bunch of photos as you move the phone around a bit and reconstructs the depth from multiple shots.

http://googleresearch.blogspot.com/2014/04/lens-blur-in-new-...

I wish these groups would make their code available so others could play with it and test it out. Some do, but I wish more did.

I would be curious to see the performance of this 3d depth reconstruction technique for non-rigid environments.

I re-implemented the techniques described here for a former employer (which sadly means I also cannot release the code). I was able to reproduce the quoted performance figures (both accuracy according to the paper's metrics and execution time), but the actual quality of the output was not very good.

Something like > 90% (number hazy based on vague recollection) of images produced results that had gross, glaring errors that would be unacceptable to a human. Errors like, "We added a horizon line where there wasn't one in the actual image," etc. But a few images produced compellingly plausible results.

It's still very interesting stuff, and some of his later work on using this as an additional prior for more normal structure-from-motion pipelines looked pretty promising. In the form reported in this paper it wasn't something ready to be productized. Although it's also completely possible that I just screwed something up.

The YouTube video in the HN comments links to one of the authors' Stanford page, which redirects to this page at Cornell: http://www.cs.cornell.edu/~asaxena/rccar/

There you can find the source for both the vision code in MatLAB, and a link to http://make3d.cs.cornell.edu/code.html for the full source.

Thanks!

Yeah! Having spent much of the last year plowing through computer vision papers, it's pretty rare to encounter published code or even datasets.

Why is this? Do people spend all their time polishing prose and neglecting the code? Or maybe they keep it closed for commercial intentions? Or is it just a cultural thing in all of academia?

Speaking from experience, the code is normally functional but unsightly, thus it would be embarrassing to place it online. Most researchers are happy to email you the code if you ask for it though.

Most of my code is for scientific computing and consists of a bunch of one-off Jupyter notebooks and Julia scripts that are written with two objectives in mind: get-the-task-done-before-my-advisor-meeting and make-it-as-fast-as-possible. Good coding practices and code cleanliness are not priorities. And particularly for code that needs to run on HPC clusters or supercomputers, it's really hard to make it look nice — one could easily spend weeks designing a proper, modular system only to throw it all away in favor of using a better algorithm that was just published. Anecdotally, the vast majority of my code has produced results that were later discarded.

For example, I have a function named YPXt. What does such a woefully named subroutine do? It multiplies a matrix Y by a permutation matrix P, and then multiplies that by another (transposed) matrix X. The matrices are small, so the overhead of using BLAS for the task is wasteful. It's better to use Julia's macro system to generate the necessary for loops on the fly and let the compiler SIMD-ify the result. Also, permutation matrix multiplies can be done in a way that doesn't actually involve creating a full permutation matrix. And of course the subroutine has side effects — in order to conserve memory I need to overwrite the input matrices instead of allocating new ones. Immutable data structures may work great for functional programming, but the overhead of red-black trees and stateless programming is too high for numerical data crunching.

Add that to the fact that a lot of scientists in academia are unfamiliar with tools like Git, and so manually versioning the code and emailing it back and forth is unfortunately a necessary part of the collaboration process.

I completely agree with you and I can relate with your "objectives in mind". But you really need to push to get others using Git.

GitHub's GUI client is very easy to use and with a bit of nagging around you can get even people who don't understand what VCS are to use it. The one time they ruin the code and get the chance to easily go back to how things were before they'll see its value and endlessly thank you for it.

In my experience, blind reviewers for peer reviewed journal articles in certain cases may argue that publicly available binary or sources are a prior publication of the technique or method, and a reason for rejecting the paper.

This risk is hard to assess in advance. Many journals do not clarify the policy regarding advance publishing of source code or even binaries alone.

Given the impact this could have on a student's career, this is worth considering carefully, and if possible, clarifying in advance with the journal's editor or program chair.

Science = Nature.

Code = Applying Nature.

This would be a lot easier to jibe with if the paper was based on proof alone. A lot of the work in this realm isn't that. Plenty of these researchers are also fine engineers and I'm sure have great code. It just seems crazy to develop a new technique so completely and stop short of releasing the code.

I think it's because the academic culture encourages this as if the code detracts from the purity of the research.

I see where you're coming from. I think the truth is actually dependent on the particular institution and particular academic.

For example, OpenRave is the result of academics open sourcing their research.

Update: Someone posted a link to the code.

I love open source, but there is a very good reason why academics generally don't release their code.

They are scientists: they are best at testing, explaining, and documenting nature.

Code is an engineer's job: making it as fast and robust as possible for the given application or with the latest tools.

They want to make you work for it :)

With all due respect, that is a stupid reason.

I don't have time to implement every last thing. And I don't require robust code all the time -- it's very understandable for the code to not be robust if it's being provided by a small group with not a lot of funding or resources. Hell, if they make it open source I will contribute to it expecting nothing in return to make it better in any way I can.

I really think that the reason the code isn't shared is journals don't encourage this type of thing (or make it easy). Lab websites live and die, so hosting it there is not a good idea. Very simply, the code should should be packed with the paper somehow. I hope some journal takes the lead soon in creating a system that encourages a bundled presentation of paper + code.

Academics aren't always concerned with making your job easy, they have lives too. They may have struggled a lot for their results and their code is still a living document, or they sell it to a company. Maybe their grant ran out and they got pulled into something else. Maybe they died.

There's a lot of factors.

Look at OpenRave. It's the result of an academic open sourcing his research.

I just think at the end of the day, a research paper exists to validate human understanding of nature - not give you a boilerplate for implementing it.

Always love seeing computer vision related stuff posted.

Incidentally, we have improved on these techniques (given that this came out almost a decade ago!) for scale reconstruction with our mobile monocular SLAM system.

The authors are now killing it worldwide (Obviously for Ng) working on applications.

tl:dr; Can combine monocular NN CV techniques with stereo techniques for good, cheap results. AKA: The future of SLAM.

I do not think this kind of feature-based estimation of depth will be the future of SLAM. Point this algorithm at a scale model or a framed photograph and everything will go haywire. It's great for scene understanding, in which you point it at a photograph or artwork and you want it to understand what's going on in the photograph. But not for mission-critical mapping and navigation.

Depth cameras, which are getting cheaper, will be the future of SLAM. SLAM algorithms for depth cameras are also a lot simpler to write. And with depth cameras, you're not estimating how far away objects are, you're actually measuring their distance. Data beats estimation.

Depth/stereo cameras are the future, I agree. But the conclusion of this paper says that you can cascade the results of this algorithm with the stereo algorithm fairly easily.

Reply on news.ycombinator.com