Eye-tracking is a missing input device for VR experiences

That sounds like a slight miscalibration of the eye tracker. Assuming perfect eye control, if the tracker estimates your gaze slightly incorrectly then the cursor will always move away when you try to look at it.

I made an eye tracking UI for my masters thesis, with a few different components. Although they all worked, having my eyes tracked to control the UI felt very intense, it caused a lot of mental and physical strain.

I'm not saying that my thesis is evidence it absolutely can't be done, but it was surprisingly uncomfortable even for what seemed like basic functionality like for example autofocusing the input boxes you are looking at. But maybe there are tweaks that can make it work, I think the main issue is the screen giving feedback on where you happen to be looking, which gives this unnatural feeling that where you look is causing side affects.

Saccades also present a hardware limitation. During rapid movements, the eye can't focus correctly. To improve the overall UX, we throw away those frames entirely, reconstruct them with a linear interpolation of the next few frames, and call it a day. It does introduce some lag, but the user-level process is usually lagging behind anyways, and doesn't notice the input lag so long as we backdate the timestamps.

The Valve Index showed that as long as your VR headset is $1000 and above, barely anyone will buy it. And that's without taking into account the hardware you need.

If anyone has a chance to succeed this generation, it's Sony with their PSVR. And even then, it's $500, and you need another $150 for their headset.

In other words, we can correctly assume the converse; Any application that involves the user moving in VR is presently a non-starter for the most part.

Now, how fun / interesting / useful are those S/T experiences, especially as compared to the hype of VR? I'd say a solid-but-small chunk of gaming at best. Right now, I feel like the MAX impact VR could have on the world would be roughly equivalent to the Wii's.

I'm not saying that nothing could flip the switch in the future, and I do acknowledge those "physical" careers that would benefit greatly, e.g. architecture. But I see nothing remotely close on the horizon that puts VR headsets in households on the level of iPhones or even Echos.

Northern Europeans have very light irises. It makes it easier to track the pupil from a single signal-to-noise perspective. But there are a ton of weird ethnic challenges for eye tracking. Things like reflectivity and moisture of eyes varies a lot across different across different ethnic groups. Personalization add complexity and will only take you so far when you are dealing with physical sensing challenges.

I've seen NVIDIA and Tobii (highly motivated parties) claim that you might get a 10% power reduction, but those studies were based on very constrained operating conditions and conveniently brush certain details under the rug. All told, it's a lot of work and a lot complexity to chase something that will at most provide modest gains.

The cameras on headsets face the world. So you need 2-4 additional cameras trained on the eyes. These camera each require power just to get images. For a decent UX you need tracking at ~120hz (60hz eye tracking is slow). Exact figures are hard to come by on a Sunday morning, but off the top of my head, I've seen figures to the tune of 250mW per camera (which ignores ISP and the compute to move the data around). Computer vision algorithms running at 120 hz are typically not cheap. Pupil Labs cites 50% CPU (on an i5) per eye[1]. Tobii doesn't publish their power consumption but it is also pretty steep. I'm sure there are optimizations here but all the specs I've seen blush at the power consumption required by a general purpose eye tracker.

If you have figures that suggest otherwise I would be happy to see but every quantitative analysis I've seen comes up with a muddled response. If you already commit to running eye tracking for other reasons then it makes sense to run this rendering optimization, but I'm not sure if it really pays for its self. [1] https://pupil-labs.com/products/vr-ar/tech-specs/

Just because you already are running cameras + ML models that doesn't mean adding even more cameras and ML models is free.

Part of the problem is you can throw everything you have at motion tracking. You can over render and then use the end of frame head position to space/time warp to adjust the render the display gets, maybe throwing away some of the edges.

For foveated rendering, you need to know what to render at the START of the frame before you render it, there can't be any waste, and we're only talking about a thin band of rendering gain IF the user is not looking forward. I would assume the focal point can change faster because you're at least dealing with the head movement + eye movement, so it's even more work that needs to catch up at the end of the frame.

I really do not think foveated rendering is easier than head tracking. If it was, we would see more of it. Dynamic foveation is really only just now coming to headsets.

I assume saving power means skipping a lot of processing, which sounds to me like undersampling i.e. the opposite of the supersampling used to get good renderings today. But, it is hard for me to imagine how a generic 3D modeled scene can be undersampled for the periphery without introducing horribly unnatural aliasing artifacts. In other words, you still need to determine proportional occlusion and lighting of all the elements in the scene to understand which blurry colors to show. You can't just cast a few rays and stretch them to fill the periphery. Otherwise, slight camera or object movements might lead to chaotic flashing of peripheral colors as the reduced samples stochastically observe different scene elements while ignoring the rest.

I know of things like multi-resolution texture pyramids which can precompute a blurred version to allow undersampled texture rendering. And I know many games switch between different low/high polygonal representations of objects (or even cull objects) based on distance to the camera, which often produces obvious "popping" artifacts as the user moves through the 3D world. Are there general scene modeling techniques that can work as smoothly as multi-resolution textures to allow rapid or simultaneous use of different levels of geometric detail in one rendering?

I think those popping artifacts would be much more distracting if tied to every eye movement rather than only with navigation across a 3D world. Crude, culling methods would be inadequate if they just drop peripheral objects that might otherwise produce the kind of peripheral image that would cause the user to change their gaze...

Every VR headset with eye tracking uses actual cameras.