How difficult is it to track eyes using modern deep learning techniques? Isn't this a simple, run of the mill task by now?
There aren't large labeled datasets that are publicly available in this area. Just to classify eye movements into events, you would need images of the eyes or eye velocity to classify eye events into smooth pursuit, fixation, etc. That means someone needs to label all of those in ground truth. To actually determine what someone is looking at, you have to go beyond this to have information about the scene. When looking at a screen, this is usually done with calibration, but things get more complicated when trying to predict what someone is looking at with a mobile eye tracker and a fixed scene camera. Body and head movements complicate things further (e.g., VOR) and many algorithms ignore them.
Video and demos and source code: http://webgazer.cs.brown.edu/
http://www.businessinsider.com/facebook-acquires-gazehawk-fo...
Eye tracking is tricky. I worked a bit with two guys developing a commercial eye tracker in the 80's, it was a pretty sophisticated optical set-up that combined the image being looked at and the image of the person looking using a 45 degree angle 50% mirror, that made the job considerably easier.
One of the complicating factors was that eye movements have a lot of noise in them that our brain has become very adept at filtering out so what we think our eyes are doing can be quite different from what they're really doing.
That task should be achievable, but anything fancier - even positioning a dot - is probably still a tall order with that single camera.
Warning sign on a friends watercooled laser: 'do not look into laser with remaining eye'.
The hardest part were ellipse curve-fitting algorithms - I had to go pretty cutting edge for that.
[0] http://blog.chewxy.com/2014/04/08/eyetracking-jetpack-joyrid...
His entire keynote is great, particularly if you're interested in the near-future possibilities of VR.
The moment the stuff you want to track (eyes) also makes the enviroment change (ui) stuff can get tricky.
Eye blinking also changes the way we perceive stuff. If we use our eyes to control a UI, blink events are also special because they usually happen when we have processed enough information from a scene (Walter Murch book: In the blink of an eye tells something about this from a filmmaker perspective).
So... not as easy as it looks at first sight (no joke intended ;-).
Identifying eyes is fairly easy, of course. Identifying open, approximately centered eyes is the easiest version of this and it's what we want.
Basic eye-tracking efforts say something like "which of these equidistant squares did your line of vision intercept?" That's still enough to produce some cool things like "look at the red box to click it", but it's not a real-world solution.
But the problem is identifying small changes and cross-referencing them with the state of the world to identify what's actually being viewed. The difference between someone looking at a sticker on a window and looking out that window is big, but the evidence available to distinguish them is tiny.
Other 'natural' movements can also spell doom for eye-tracking: turning your head without moving your eyes sweeps your vision, while turning your eyes saccades it and only values the endpoints. And turning your head and body while keeping focused on a single point feels simple, but identifying it means tracking three different axes of rotation as they interact!