If someone was given a similar device at an early age that was semi-permanently attached to them, would their brain possibly be able to create a map of the room?
There have been previous attempts but the Tango device didn't exist then so the hardware was bulky and usually required a backpack.
There has been some prior work on using depth cameras for navigation for the visually impaired. For example, a smart cane can detect objects beyond its reach and give haptic feedback. Microsoft Research did some work with putting the Kinect on a helmet and giving audio cues for navigation (http://research.microsoft.com/pubs/184208/VisionForTheBlind....). What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it.
One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. Maybe if someone was using it from birth, but for relatively fast training I doubt that approach. Other approaches do more interpretation -- Microsoft's work detected faces, walls, and floors, giving each a distinct sound for greater recognition.
Have you tested this approach with blind users? I think building a picture of an environment is a good task to offload to the brain and a good skill to have/develop for blind people.
> One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise.
I think this is a quite good approach, but agree it has a high learning curve. However, that high learning curve might reward the end-user with a system that is more flexible. By preprocessing the input and generating audio based on the detected patterns you limit the applicability of such a system. That being said, a generic system that gives "unfiltered" output and has additional cues you can set for example for fast approaching objects might be useful.
Another similar project lets people "see" with their tongues [2]
I definitely think using binaural (3d) audio could give users a much more complete and useful idea of what they are seeing so I wish you luck. Great Idea.
[0] http://www.ted.com/talks/david_eagleman_can_we_create_new_se...
[1] http://www.eagleman.com/research/sensory-substitution
[2] http://www.scientificamerican.com/article/device-lets-blind-...
A human generated "click" is much different than a computer generated series of sounds which represent an accurate scan of the objects in front of the user.
Change in pitch representing changes in depth is much more able to be processed by the human brain verses trying to hear how the sound waves bounce off of objects.
Plus, this will work in public while I'd guess that the human clicking noises require a quiet environment and have significant limitations.
Imagine a blind person walking freely down the sidewalk and the device would not be making any sounds until there is a sign or building within 20 feet, with a scanning (or single point) tone that gets progressively louder. Effortless echolocation.
The device also has several GPS, WiFi, gyroscope, and cellular geolocation abilities so it would know when the user has reached the end of the sidewalk, if outside.
I remember this story from a few years ago: https://www.youtube.com/watch?v=YBv79LKfMt4