Google buys Eyefluence eye-tracking startup
techcrunch.com
techcrunch.com
The two main cases I'm thinking of are eye tracking, and brainwave tracking (like EMOTIV). Individually, neither looks like a compelling way to control a computer when you have a mouse and keyboard, but together, I think that might really yield something interesting (and sooner!).
Instead of using brain waves to move a cursor around the screen, use eye tracking. Instead of using gaze lingering for click, use brainwave tracking. Individually they seem cumbersome and annoying to use. Together they could be a really compelling interface device, IMO.
Eye movements cause travelling waves to sweep across visual brain areas (which are actually a huge chunk of the brain), potentially swamping whatever was going on before. Some of this is probably supports visual processing (by, e.g., persisting or resetting neural activity across eye movements), and some of it might be epi-phenomenal.
Furthermore, EEG electrodes pick up huge, non-neural signals from the movement of the ocular muscles and the movement of the eyeball itself. This can be reduced with clever pre-processing and more expensive amplifiers (e.g., that don't saturate or recover from saturation more quickly).
There are many big problems in VR that get solved if you know where the user is looking at. The most important one is, your eyes really only see high resolution in a very small region around what you're staring at, and if the VR headset can only really render that region in high quality, this solves many issues (processing power, framerate, even wireless transmission).
Of course, for any of this to work you need perfect eye tracking, which is very hard.
Here is a video of a 2016 demo of eyefluence tech:
(Serious version of this: Amazon could feed eye-tracking data into their "stock a product near the user if they're likely to buy it" algorithm.)
Husband: Nope, sorry, no idea!
What does sound like a problem is eyes for something they're not intended for, being hands and fingers, I wonder if people will end up with RS'eye' ;)
Also your solution seems... excessive...
Whether you do that or not, use an OS that you trust to mute the microphone, and don't run untrusted applications outside a sandbox.
Privacy aside, I have an ugly face.
" . . . I'm still not allowed to say how exactly it works but in my mind it's the gold standard for eye based user interfaces . . . Their [main focus] is the UI" (1)
So sounds like a blend between some good eye tracking tech and some GREAT user experience design and implementation.
(1) https://www.reddit.com/r/oculus/comments/598clo/google_buys_...
There aren't large labeled datasets that are publicly available in this area. Just to classify eye movements into events, you would need images of the eyes or eye velocity to classify eye events into smooth pursuit, fixation, etc. That means someone needs to label all of those in ground truth. To actually determine what someone is looking at, you have to go beyond this to have information about the scene. When looking at a screen, this is usually done with calibration, but things get more complicated when trying to predict what someone is looking at with a mobile eye tracker and a fixed scene camera. Body and head movements complicate things further (e.g., VOR) and many algorithms ignore them.
Video and demos and source code: http://webgazer.cs.brown.edu/
Warning sign on a friends watercooled laser: 'do not look into laser with remaining eye'.
http://www.businessinsider.com/facebook-acquires-gazehawk-fo...
Eye tracking is tricky. I worked a bit with two guys developing a commercial eye tracker in the 80's, it was a pretty sophisticated optical set-up that combined the image being looked at and the image of the person looking using a 45 degree angle 50% mirror, that made the job considerably easier.
One of the complicating factors was that eye movements have a lot of noise in them that our brain has become very adept at filtering out so what we think our eyes are doing can be quite different from what they're really doing.
That task should be achievable, but anything fancier - even positioning a dot - is probably still a tall order with that single camera.
The moment the stuff you want to track (eyes) also makes the enviroment change (ui) stuff can get tricky.
Eye blinking also changes the way we perceive stuff. If we use our eyes to control a UI, blink events are also special because they usually happen when we have processed enough information from a scene (Walter Murch book: In the blink of an eye tells something about this from a filmmaker perspective).
So... not as easy as it looks at first sight (no joke intended ;-).
The hardest part were ellipse curve-fitting algorithms - I had to go pretty cutting edge for that.
[0] http://blog.chewxy.com/2014/04/08/eyetracking-jetpack-joyrid...
His entire keynote is great, particularly if you're interested in the near-future possibilities of VR.
Identifying eyes is fairly easy, of course. Identifying open, approximately centered eyes is the easiest version of this and it's what we want.
Basic eye-tracking efforts say something like "which of these equidistant squares did your line of vision intercept?" That's still enough to produce some cool things like "look at the red box to click it", but it's not a real-world solution.
But the problem is identifying small changes and cross-referencing them with the state of the world to identify what's actually being viewed. The difference between someone looking at a sticker on a window and looking out that window is big, but the evidence available to distinguish them is tiny.
Other 'natural' movements can also spell doom for eye-tracking: turning your head without moving your eyes sweeps your vision, while turning your eyes saccades it and only values the endpoints. And turning your head and body while keeping focused on a single point feels simple, but identifying it means tracking three different axes of rotation as they interact!
At the time, i read it was only a acquihire
They'd be worth acquiring if only for the marketing department.
Much smarter than investing in VR.