Structure Sensor: Capture the World in 3D
kickstarter.com
kickstarter.com
Loads of people walking around with even better 3D sensors stuck somewheres on their head. The devices are networked (a feature which has shown the ability to save lives in a number of tense evacuation conditions) and upload to a common database. Algorithms have sufficiently numerous snippets to calculate optimal stitching. With haptic VR you really can go to a place without leaving your home. Clever people are also devising agents that go out and calculate optimal itineraries for busy travelers. There is a market in there to keep top routes relatively clear.
A handful of companies vie to be gatekeepers to this data. I think Google is amongst them. In some timelines people are micro-paid cataloguers and explorers. Most people get a tiny amount for just walking around and sharing. Some teams are very skilled at this and get paid well for their unique 3dgrapher experiences. Tombs and such had labor supply so high, they were cheaply and quickly mapped out. Level design became a fair bit easier. Mostly boring places left to map. So dull, pay is decent for walking around thoroughly in them.
The 3D sensors are good enough that facial recognition beats anything a human can do. People get micro-currency kickbacks if they let their expression geometry and infradata be tracked as they visit various stores. The prior mentioned gatekeepers use location and networked devices to track/monitor your face and infer emotions as you browse assorted merchandise. Main individual utility is mostly that no one ever gets lost anymore and all users know where everything is. No matter this is their first visit. Government expands funding into denser, more reliable storage. Everyone benefits. Laws really need to take into account that as soon as you leave the house, your every step is tracked. Everyone is hence at least a minor criminal.
Laptops and such are also fitted with similar more precise sensors so Ninja Bull Growth Hackers can track every minute subsecond flicker of expression, every heartbeat and temperature fluctuation as you browse whatever the current iteration of social media photo sharing is. Opt-in by default, setting up micropayments is a tad tricky.
If we just take Apple as an example, for them to include it standard, it would have to really justify its existence. (I realize many people don't care about Apple, but I choose Apple because, for example, there have been stereo cameras on various Android models over the years, but since developers can't really count on the presence of a stereo camera for a large number of users, nothing game-changing ever really came of it, even though stereo cameras enable 3D sensing.)
Though user interaction and 3D acquisition are amazingly cool, I don't think Apple would ever include a depth sensor just for those reasons. However, Apple has shown it cares pretty deeply about the iPhone as a camera and is even willing to add new hardware if it helps people take better photos, the dual LED flash being the latest example of this.
So, as crazy as it sounds, I think Apple could add a back-facing depth sensor in 2-3 years mostly in order to allow for automatic depth-based post-processing of photos, mostly for better-looking lighting, all without the user having to care that there was 3D sensing involved. All the other stuff will then become possible, and widespread, as a side-effect.
EDIT: On the Android front, the PrimeSense folks had a booth set up at CVPR this year and mentioned that at least one manufacturer was planning to include their Capri sensor (a miniature Kinect) in an Android tablet some time soon.
PS -- Hi Grant! From a fellow GT Robotics alum.
I also like the Oculus Rift hack that Occipital showed briefly here -- it makes a lot of sense for augmented/virtual reality, though I guess it means you can only interact with things when looking at your hands, as opposed to Sixense's controller-based approach.
PS: Hi, Travis!
- Games. Look at the tennis ball example on the original article. Remember Google's IRL photo-based game (Ingress), now just imagine that with fine-grained 3D (Google will use it to crowd-source a centimeter-model of the earth). Games like this remain a bit of a niche, but just imagine if someone makes a massive social game out it on FB (a cross between Farmville, Sims, and Minecraft, projected onto your real world). Of course, someone could also create a shocking IRL FPS game (imagine your kids pointing this out the window in traffic and "shooting" at people and cars to watch them blow up). Finally something to use the processing power in theses little phones and tablets.
- 3D photography. I think this is the future of photography. Take a picture of something, extract the spacial data from the image, modify it/change the p.o.v. Recall the recent image/object maniupulation video (the SIGGRAPH one that used the PatchMatch algorithm to fix the background). Each photo becomes a mini-scene that you can navigate around (kinda like the "frozen" 360 degree pans in the Matrix). Next step is time-dimension, in other words 3D immersive movies where the viewer can move around almost anywhere while the movie unfolds. You can guess the first industry to adopt this...
Both of these can currently be done with flat images, processing power, and some human guidance. With depth sensing it can be faster, automated, and more accurate.
Hm. I now envision a (physical) addon that has a two mirrors, like a periscope, and fits over the end of the phone, and the "face" camera -- allowing the two cameras to be used for stereo imaging... Not sure if you'd be able to do just as well with motion sensor data coupled with just the main camera -- and having the user move the phone from side to side (using parallax for depth sensing) ... would make for an interesting algorithmic problem at least.
How often do you need to scan a 3D model of anything?
I'm sure it will be popular with the creative people. It's possible it will get popular in some niche markets, like real estate agents taking 3D shots of the apartments they offer (though not much hope here, they still fail to take decent photos even now).
If (when) we can make a cheap and small sensor that can provide depth data to the camera to make cool looking 3D pictures, it'll likely find its way on smartphone cameras.
In 2050, will our pictures still be 2D arrays of RGB pixels?
Every day.
The 3D scanning technology would enable products that are add-ons to objects where the original manufacturer did not see value in adding an interface. If I can scan my car interior, I can attach my GPS to nearly any surface "perfectly". This gives aftermarket products very nice finish with a much lower cost.
I can see this dramatically reducing the amount of time it takes to build 3D environments for games once the tech has had time to mature.
Imagine fitting a fleet of drones with this tech and sending them out exploring...
When I saw this submission, this was the first thing I thought of: http://www.youtube.com/watch?v=IxmJT5xT5rQ
From related 3D scanning videos, it looks like 3D scanning is already a pretty common operation, e.g. in mining and navigation. Putting the technology into consumer-facing mobile devices... wow.
There is a good tutorial on it at http://www.blenderguru.com/videos/the-secrets-of-realistic-t...
It is a program that takes a single image and generates diffuse/normal/occlusion/specular/displacement textures from it in such a way as can be easily used as a material in blender.
As you can see from the video, it can work amazingly well given the limited amount of input information.
Obviously that's something that involves multiple stages, and I can think of three. The first is removing gradients from diffuse shading. The second is lightening shadows to match their surroundings (shadows, incidentally, could probably be used in combination with the 3D structure to infer light sources). Finally, you'd need to identify specular highlights and inpaint them. You might also have to use inpainting in stage 2, in order to deal with full-black shadows.
I see this sensor as something that has an immense number of valuable uses to everyone, as well as creating an immense number of potential abuses.
Whenever I see new technology I think of the Mom and Pop shops, run by people who aren't a part of the technology movement and how it would effect them. And when I think of that shop and this device, I think of people being able to scan the store and not just to get a video image of the layout and camera locations, but a properly rendered floor plan of the store where one could pinpoint unsurveilled merchandise.
I also see a lot of regulation coming from this. Imagine a world where anyone can go to a store, pick up an object, scan it at the store and then go home and print it on a 3-d printer. It might require theft to be redefined and create a slippery slope for people who like to take pictures of things with their iPads for creative inspiration.
I'm not trying to argue against this technology by any means, but with every new technology that comes out, a lot of questions are raised for me.
I guess the issue for me is more focused towards humanity rather than the technology. It's probably a nonsensical concern as there's no way to know if we'd really be better or worse off with/without any technology that has been developed and disseminated. But like Uncle Ben/Voltaire said, ~"With great power comes great responsibility."
With the advancements in technology and their increasing acceleration, we are being spoon fed an increasing amount of power. My concern is whether or not we are responsible enough for it. This sensor looks amazing, and I can't wait to see the great uses for it but like every new technology that completely amazes me, it reminds me: The future freaks me out.
Brick and mortar stores will cease to exist long before this technology becomes good enough to do what you've described. This technology is currently pretty awful in terms of accuracy. Btw, what would you actually be printing? A plunger? Easier to 3d model. A clock? You'll never measure it accurately enough. Some kind of cheap coat hook? Not sure the store is surviving off of cheap coat hook margins in the first place.
Btw, these kinect-like cameras work by measuring the displacement of infrared dots, so there exists huge discrepencies in first the infrared dot projector, then the camera that sees them, and also the surface onto which the dots are reflecting. If you're lucky enough to get decent input data, then you have to post-process the noise out, so that flat surfaces are indeed flat, and curved surfaces have a smooth curve. But at what point is noise actually just subtle features? Imagine trying to scan a diamond-plate floor panel. The diamond design pattern would be perceived as noise, and the optimizations would attempt to normalize the data to be a flat surface.
There's a long way to go before we have to be considering the dangerous power of this kind of tech. The current danger is in 3d printed firearms, which are getting better and better every day, and which have been 3d modelled by designers in 3d programs.
> The current danger is in 3d printed firearms, which are getting better and better every day, and which have been 3d modelled by designers in 3d programs.
It was probably discussed here many times, but where exactly is the danger? You can (and could have for years) make better guns with CNC milling machines and/or random metal scrap lying around your backyard, and getting ammo (which seems to be the real problem) is not getting any easier with 3D printers.
I bet you could get around this somewhat if you matched the 3D data up with a normal 2D image, processing it to figure out textures and subtle features.
Not to mention that if you went linux/android you'd probably also support desktop for "free" after implemeting this over USB and writing the kernel driver.
However this video shows that their 3d reconstruction is actually pretty good:
https://www.youtube.com/watch?v=w4aMQQv2Zvk
I think that an open platform like theirs, which already includes sample software that does good 3d reconstruction, is very good for the field. Much more interesting than projects that combine hardware and software so that only the makers can experiment with new software algorithms.
EDIT: another video showing it "really working"
Hopefully this gets funded, looking forward to getting my hands one one!
http://us.store.creative.com/Creative-Senz3D-Depth-and-Gestu...
I imagine that will be quickly hacked to do much of what this structure sensor wants to do.
Edit: Date, they slipped from the 25th.
From the video:"Structure works by capturing a pattern of invisible, laser projected light, which allows you to measure thousands of distances, all at once."