Hacker News Reader

Comment by SilverBirch | Hacker News Reader

This is quite interesting actually, because this sort of directly lays out "Here are 3 difficult technical problems we need to solve for a VR headset" - Resoution, Focal depth, high dynamic range.

I'm not an expert in the area, but resolution and HDR seem like basically solved problems - in that they're just logical progressions of where we are today. The focal depth one I didn't understand. He says normal monitors are a fixed distance, whereas in VR and AR you need to focus on different distances. But these VR headsets are just a fixed distance away, so how is that really a problem?

Fundamentally these problems are clearly necessary buticie not sufficient for VR.

View on HN

When you look at a monitor, you're looking at a quad at a set distance. Your eyes are focused on that quad in the exact same way they would focus on anything at that distance.

In VR, dynamic depth is simulated using stereo-screens where each pupil is pointed at a dynamic focal point BUT stero-focus is not the same as lens focus. Because of this, VR produces a disjoint sensation where stereo focus changes to the simulated position but lens focus remains fixed.

You can experience the difference by holding up a finger and looking at it, then look at a distant object. Notice that you'll see two images of your finger as you focus away. That is stereo focus. Now do the same while covering one eye. Notice that the finger is now blurry but not doubled. That would be the lens focal difference.

Ah yeah, I had forgotten about the issue of presenting separate views to each eye, good point.

Your brain puts together multiple 3-d cues. If you are looking directly at something the angle of your two eyeballs is a little different dependening on the distance and this is vergence.

Your eyes also focus like the autofocus of a camera and the cue from that is called accommodation.

The two should match to provide perfect perception on reality. Certainly a VR headset works with a fixed focus for everything, but to get the ultimate perception of reality without eye strain a VR headset should be able to simulate focusing distance.

(Who knows, however? Meta’s Super Bowl ad might be revealing their real intentions. In that ad a discarded animatronic Android gets to relive its past with VR. VR is good for the elderly because you can enjoy it without learning anything new. I think one of the worst things about getting old that I experience is presbyopia where you can’t focus over the whole range so you have to wear two pairs of glasses. Maybe I’d find it easier just to have it all in focus all the time.)

> VR is good for the elderly because you can enjoy it without learning anything new

I hope this will eventually be the case, but with current iterations, navigating and interacting with VR UIs and experiences is quite difficult for people who aren’t already familiar with game controllers and console-like menu systems - especially the elderly.

You're right to point out that controls should be on the list of major problems required for VR to be mainstream.

On focal depth: In the real world you can look at an object close up and your eyes will adjust so that it is clear and objects at other ranges are blurry. Then, when you look at an object far away your eyes re-adjust focus.

You can test this by looking at your hand 6” from your face so it partially blocks your keyboard a couple feet away. You’ll notice that either the keys are blurry or your hand is as you shift focus between the two.

Future gen headsets will use eye tracking to understand which object in a scene you are looking at, and make that object sharp while making other objects blurry. This helps produce more realistic depth, while also dramatically improving performance as most of the scene can be rendered in lower resolution.

This is key to me. I have a quest 2 and I don't like having to move my head instead of my eyes.

The screen being a fixed distance is the issue. The holy grail would be a system that fools your eyes and makes you think that it isn't at a fixed distance- something like eyeball tracking that detects, instantly, what you're focusing on, and adjusts the perceived focal distance based on how far away the cluster of pixels you're looking at is meant to be. This would improve immersion.

HDR is trickier than you think because devices like cell phones can improve their dynamic range by just making the screens brighter- increasing the range by raising the top end- but there's a certain cutoff on how bright a VR screen can be and still be comfortable.

I'd throw in 2 more, power/heat and weight.

However, the resolution/refresh-rate needed for immersive VR/MR is not quite a solved problem. If you assume something like 100deg horizontal and vertical for each eye and something like retina (not screen door or blurry) 40-60pix/deg resolution, you're looking at 5k x 5k per eye at 120-180Hz for 2 eyes. You can't do that over a single DP 2.0 link, and it would be too power hungry anyway. That leads to a requirement for fast eye-tracking and foveal rendering (only rapidly refresh where you're looking in high resolution)... and gains you ~10x reduction in bandwidth/power.

Then you get to directly monitor the user's attention, build a DL model of their attention, optimize it for maximum interaction, and sell the model to the highest bidder.

> But these VR headsets are just a fixed distance away, so how is that really a problem?

Focal depth is one of the cues your brain uses to perceive distance, in addition to (potentially more than, depending on which cognitive scientist you listen to) binocular vision. You don't mind that monitors are a fixed distance from your eyes because you don't expect them to give you real depth (your eyes can just focus on that distance). If, however, you want something to be "indistinguishable from reality" you need to emulate changing focal depth, which means (I guess) changing the angles that rays hit your eyeballs at.

IMO that's one of the reasons that 3D movies always looked so fakey; they could emulate the binocular vision, but they couldn't emulate the focal depth, causing a perceptual dissonance.

There are multiple ways that your eye/brain senses depth: binocular vision/vergence, lens focus, and relative correlated motion to name three. When these depth cues don't match each other well, it is distracting and can cause fatigue or headaches after extended use.

The focal depth thing is related to a human vision system bug/feature - Your eyes want to change their 'vergence' at the same time they change focal distance. This is sort of a hard-coded geometry solution. When you want something close your brain crosses your eyes a bit, adjusts the image to your brain, and changes focus. When you look far away your eyes uncross a bit, apply another transform to the result, and change focus again. Getting the eye-crossing and simultaneously trying not to change focus is one of the things that gives people eye strain and headaches when using VR.

> this sort of directly lays out "Here are 3 difficult technical problems we need to solve for a VR headset"

It lays out the problems Meta had most progress in. Another very significant VR metric is FOV which was not discussed.

> He says normal monitors are a fixed distance, whereas in VR and AR you need to focus on different distances. But these VR headsets are just a fixed distance away, so how is that really a problem?

You want dynamic focus to convey the feeling of real world eye focus, and make the projected scene more natural/believable.

Solved but still not enough to make it seem reel. He said that natural light has 10x more dynamic range than best monitors available. As to focal depth - in real world there is no "screen", your eyes (or perhaps brain:) can decide which what you want focus on and what can stay blurred. On screen everything is in focus, so you need to make fake blur. But you need to know what user is focusing on. So you need to read retina movements to guess. Sounds like complicated and hard problem to me.

I actually think the thing that is going to make VR mainstream is more comfort from going to goggles or glasses rather than headsets. Or even better, something you can attach to glasses.

> The focal depth one I didn't understand.

I think the idea is simulating depth of field by blurring different parts of the image based on where the user's eye is looking.

Reply on news.ycombinator.com