Hacker News Reader

Eye tracking does not work nearly as well as people imagine. It cannot directly replace a mouse pointer the way you want it to. The accuracy and reliability are not good enough and never will be due to physical constraints. This system is likely already working better than an eye tracker would for cursor control, and it will certainly improve.

Apple has done great stuff with eye tracking on Vision Pro, but it required completely rewriting the UI for literally everything. Not something we have the luxury of doing for accessibility for quadriplegics.

Source: built an eye tracker and eye-controlled UI at a startup and got acquired by Google

I hear you. I literally cant use any commercial tracker because of a alternating strabismus and the midas touch effect is a nightmare. I'd argue Apple did that because its such a mess on regular OS where the primary input method isnt eye gaze. (They did that with iOS too - match the OS and UI to the primary access method)

I'm not sure I can believe this "This system is likely already working better than an eye tracker would for cursor control" - the training to use this stuff isnt just 'magic'. I do agree though with "and it will certainly improve" - yeah iterate.. iterate..

Eye tracking on AVP works very well on websites that are not at all designed for it.

He notes these other forms of HCI in the video, and I think you’re really underselling the main point of this being ease of use. All of those methods are significantly degraded experiences compared to normal ways of interacting with a computer. The potential for a quadriplegic to interface with a computer at a higher ease of use compared to a human without disabilities is huge.

Sure. The problem is this video somewhat misses how that interaction actually works. You need a control technique of thinking to action. What that actually is and how you learn to use that is missing from the demos. Thats a tad misleading right now IMHO.. (NB: It could be intentional speech eg "Move x to y" or it could be intentional motor control "move my hand right and up" You would also need a stop command. Not sure.. )

But sure - "the potential" is the key thing in all this. Just the cost to benefit ratio is pretty dramatic right now.

I don't think the point for him was to simply find a way to control his mouse. He said at the end of the clip he specifically wanted to help out with Neuralink. He also said the surgery was easy and he was released a day later.

He appears to just think about where the mouse should go and then be able to click and click-and-hold. Seems like multiple inputs which an eye tracker wouldn't do. Unless maybe its just configured to click when the cursor pauses on a spot?

Also the user experience seems better than attaching electrodes to your head. It seems to just work wirelessly. It is always there and sometimes he has to recharge it.

Theres a lot of patients who "want to help out" Eddie Chang's team in SF have one patient who moved right next to the hospital to equally do this.. (NB: they should be getting paid too by the way - although thats not clear)

Yeah - this is wireless. Better than some systems which have been, no joke, a box at the top of your head with a HDMI cable in it..

> He appears to just think about where the mouse should go and then be able to click and click-and-hold. Seems like multiple inputs which an eye tracker wouldn't do. Unless maybe its just configured to click when the cursor pauses on a spot?

This is really the key question. The dwell technique you note is what most eyetrackers do (although far better to use a binary technique - eg a blink - to select because of the midas touch effect). Its built into to Windows/MacOS and iOS now. I have a sneaky feeling the reason why its chess is you can encode the positions "X1 to Y2". You can then do a transformer model to decode intentional speech..

If that is the case - then if a person actually speaks whats the benefit right now for this indvidual? (yes - that he doesnt have to say it. BUT sub-vocal speech is already achievable without invasive surgery..)

> You can do it with surface EEG. sticking it inside your head - closer to centres where you can measure intentional and thought makes vy signal cleaner and more reliable.

Further clarification: when doing conventional EEG, the signal quality is so fragile that even blinking can produce recording artifact.

Also, there's the whole "put a shower cap with conductive gel" things that makes it very impractical for every day use.

You don't need a full EEG to do what the parent mentions. 3 electrodes around each eye (EOG) + ground is all that is needed. Even cheap commercial electrodes will work for this. Blinks can be filtered online with PCA-like techniques.

There are 'dry' EEG electrodes which don't require conductive gel, and algorithms to remove the electrical effect of eye-blinks (ICA, etc.)

I think once this tech can be used to connect to a robot / exoskeleton, then it will be very useful for someone like him. Imagine him thinking that he wants an apple from the fridge in another room and the robot goes and grabs it for him.

Yeah I agree. In reality though - right now - I'd say you might be surprised to what people actually want and need. For example I could assess someone for say a curtain opener or door opener. I have fallen into the trap of assuming that someone would want this. In reality though its very few people who do. Why? Its totally overkill and pointless when 90% they are with someone who will just open the door anyway. (NB: This isnt Everyone - the ubiquity of home control systems on the regular commercial marketplace is changing this I think)

Sorry but what you're saying simply is not true from my experience. The potential of this technology is the ability to perform multiple actions simultaneously e.g. having your character strafe while zooming in while firing a trigger. With eye tracking you are limited to (for the most part) one action at a time, which is what I do now.

You are correct in that one could add more inputs but that only works if you can use the inputs. The individual in the video has full control of his head which many people do not. All I can do, for example, is use like two fingers.

He explicitly mentions eye tracking and how it doesn't work for him in this interview with him: https://twitter.com/ModdedQuad/status/1771298116719002100

There’s a ton more that these implants can do other than cursor control. And not needing external visible hardware like other forms of input is a bigger deal than you might think, and especially in terms of the user wanting to feel more “normal” and less of a robot in a bundle of contraptions

They are taking an iterative approach.

Look at the starship program for an example of where you can get in 20 iterations