Let's Build a Superpower: see within and behind objects

We merely want to make a proof of concept that inspires users with a mind-opening sense that the world's data can have higher dimensionality than just 2 dimensions.

I (Jon) read the novel Flatland when I was a kid, and have ever since been yearning to see in more than just 2 dimensions. This project is an attempt to open users eyes in the same way.

--

With the goal defined as proving a concept, it becomes apparent that we do not need to permanently solve every challenge with AR in a scalable way.

We merely need to kludge around them (for example using tons of computing power) enough to make a convincing temporary experience:

1. Patterns of marks or lights can be placed on the outside of the headset, detectable by cameras in the environment.

2. True. Luckily we can achieve a kludgy demo by having one entire rendering computer (with graphics card) per depth slice, and selecting between their outputs as a dumb switch, post-rendering.

3. The plenoptic camera from lytro.com should provide the necessary post-capture-focusable images. We are meeting with Prof. Marc Levoy today to ask about partnering.

--

We merely hope that once the concept of voxel vision has been proven, that all future AR systems will feature it as a standard mode of interaction.

I think that in the future, the following phenomena are likely to proliferate:

- high-res satellite and airplane imaging of the globe

- public geotagged photos, including compass bearing

- Google Street View road images

- public security cameras on public streets

All of this data can be integrated using software like Microsoft's PhotoSynth, to stitch together a gigantic 3D model of the real globe.

Then, people with Voxel Vision will be able to look around at streets and landmarks that are interesting but physically occluded.

======

Edited to rise a relevant comment from further down the thread. Original answer follows:

Jon Rodriguez here, project lead.

Your question seems ambiguously worded.

I will answer both possible interpretations of the question:

===

Q:

"How do you propose to do the sensing task of seeing inside real world objects so that that data can be rendered?"

A:

The source of content depends on the application.

1: put a virtual screen at distance infinity, so in a boring meeting you just look far away to check your email

2: for telepresence surgery, use a MRI to do the 3D imaging of the patient's interior

3: Google Street View images allow you to see to any point in a city, even if buildings are between you and it

4: educators create custom 3D models of physical phenomena they want the kids to vividly explore, such as a 3D model of a human neuron, or a 3D geological model of the Earth, or the interior of a grape.

Etc.

In particular, virtual reality games and worlds can be easily shown this way, as long as the game artists take the extra effort to give their models interesting interiors.

===

Q:

"How does a Heads Up Display that is in front of a real-world object (like a brick wall) then project a virtual image that appears to be behind that object?"

A:

This is where the adjustable lens on the virtual retinal display comes in.

By properly decollimating the light from the VRD's screen or laser, you can give that light an arbitrary "equivalent optical distance".

Despite the fact that the light is originating at 2 cm from the eye, it can appear to originate at 2 km from the eye, in that 2 km is that distance that the eye will have to accommodate to to put that light in focus.