Software converts 360 video into 3D model for VR
360rumors.com
360rumors.com
I was recently trying to understand what the state of the art was here, and was surprised to learn that this is true: most 3D reconstruction fares better with a smaller number of high-resolution still photos than with a larger number of lower resolution "stills" extracted from video. I'm still somewhat confused why this is the case.
My limited understanding is that main advantage of working with stills is that they more commonly contain location tagging, while frames from video do not. Compared to the base level of difficulty of building a accurate 3D model from lossy 2D sources, figuring out the trajectory of camera doesn't seem too hard. Once one has this, wouldn't the video be just as easier to work with?
Trying to figure this out this discrepancy, I got the impression that many researchers may actually be trying to solve the harder problem of trying to compute the camera trajectory in real time, as might be needed for a self-driving vehicle: http://webdiis.unizar.es/~raulmur/orbslam/. This indeed does seem harder, but still leaves me wondering why a multipass approach wouldn't be feasible.
What am I missing? Why can't one do a first pass to calculate camera trajectory, possibly a second pass to combine temporally adjacent frames for greater resolution, then create a better model from the resulting wealth of data? Alternatively stated, why does the quality of the 3D model seem to depend more on the resolution and quality of the input 2D images rather than on the number of these images?
I believe the main problem is knowing which data to trust and which data to discard. Reflections (water, shiny surfaces), moving objects (leaves in the wind), over-exposure, and lens flares are already pretty hard to deal with. But with low resolution data, you make it even more difficult because there's more data to discard for being too inaccurate.
It really isn't a problem to use video frames, but they have different types of error/noise to high res stills. For example, the video camera is moving, and it has rolling shutter, so every pixel (or pixel row) is from a different orientation. If this isn't accounted for it introduces noise, but doing so takes more/better processing. But video frames have consistent intre-frame delay and good coherence, so it is easier to match features frame-by-frame than from a heap of individual photos. If you have good (hot GPS) positions then that makes both situations easier.
Model visual quality is improved by better resolution pictures in a fairly straightforward way... But you can get the same data by physically moving a lower res camera closer to all surfaces. It is just harder to scan quickly and maintain track if you have to do that. Some people even put lenses in front of their depth camera to zoom in tighter, for capturing finer detail.
Even just combining multiple still photos to produce a higher resolution photo is itself a hard problem in the field of super-resolution imaging. Google has a consumer product for it (mostly for removing glare): https://www.youtube.com/watch?v=MEyDt0DNjWU&feature=youtu.be....
We have used this in conjunction with screen captures from google earth to regenerate environments
Honest question because I keep up with this field and this seemed pretty novel to me (at least the quasi-DIY aspect of it)
I played around with it around 2009 or so and at that time there was scant open-source libs for bits and nothing to join it together and nothing that worked particularly well; it was the domain of high-quality high-price niche products aimed at Hollywood which you could watch product videos of on YouTube but not actually afford to use.
I don't know how accessible it is now; maybe there are now working quality open-source libs and we can all start putting it into raspberry pi robots? ;)
There were a bunch of other papers around the time.
I was downvoted a bunch here - which is odd as this really would have been news in 2005 when this would have been considered state of the art.
Right now there is nothing here which wasn't published already a decade ago.
Disclosure: I've been working on structure from motion software for the last decade.
I downvoted your initial comment, but because I thought it was unhelpful rather than because I thought it was untrue. By contrast, I upvoted your more recent comment that mentions your expertise and defends your view with a useful link to decade old software.
Still, I'd guess that for many people outside your field, "would have been news" is not the same as with "published already a decade ago". I'd guess the majority are interested in what's currently achievable using off-the-shelf hardware and ready-to-run software, and aren't bothered that it may be weak in theoretical advances. Alternatively phrased, people may consider the performance and availability newsworthy even if the theory isn't cutting edge.
That said, I'm familiar with neither the state of the art nor the state of the theory. Are you saying that you could strap the same consumer camera rig to your head, take an unplanned stroll through a forest or city, and achieve the same model quality by running the resulting video through Bundler? If so, you would have a strong case that the parent article is accepting the hype of the press release a little too easily.
The author of the article did not do due dilligence on the subject.
Until you look down. Or until you have a different eye distance than average. Or you try to move a bit and there are things close enough to make you dizzy.
Cardboard didn't need any of those fancy things, of course. It was bad enough with abysmal latency.