Show HN: Eyesite – Experimental website combining computer vision and web design | Hacker News Reader

Show HN: Eyesite – Experimental website combining computer vision and web design

blog.andykhau.com

138 pointsakchro25 comments

I wanted Apple Vision Pros, but I don’t have $3,500 in my back pocket. So I made Apple Vision Pros at home.

This was just a fun little project I made. Currently, the website doesn't work on screens less than 1200x728 (Sorry mobile users!) It also might struggle on lower end devices.

For best results, have a webcam pointing right at you. I tested my website with a MacBook camera.

Any comments, questions, or suggestions are greatly appreciated!

blog: https://blog.andykhau.com/blog/eyesite

check it out: https://eyesite.andykhau.com/

github: https://github.com/akchro/eyesite

Open articleView on HN

Nice to see people getting interested in eye gaze. There are two things that you might like to look at that can help the UX.

1 - Calibration. Looking at static dots is BORING. The best idea I've seen is Tobii's gaming calibration where you look at dots to make them wobble and pop. This makes the whole process feel like a game, even when you've done it a hundred times before. I would love to see more ideas in this space to give a much more natural-feeling calibration process - even better if you can improve the calibration over time with a feedback loop, when users interact with an element.

2 - Gaze feedback. You are absolutely right that seeing a small, inaccurate and jumpy dot does more harm than good. Again, Tobii have led the way with their 'ghost overlay' for streamers.

For an example, see the following video. After calibration the ghost overlay is used to give approximate feedback. This is enough that some naive users are able to make small adjustments to a constant calibration error, or at least give feedback that the gaze is wrong, not that the UI is not responding.

https://youtu.be/mgQY4dL-09E?feature=shared&t=36

Thank you for the feedback!

1 - I experimented with some calibration involving staring at a point, but I found it troublesome as blinking would make lead to some inaccurate calibration data (webgazer doesn't have blink detection). It was also a little more fatiguing since the user would have to really focus on staring the entire time. I found that it was less mentally fatiguing if the user could control their own calibration give themselves room to blink or just rest their eyes for a second.

2 - Ghost overlays is a really good idea. I'll see what I can do to implement that feature.

I really appreciate you taking your time to write this!

I have been experimenting with using 5 phases of movements with each phase covering different areas of the screen while being actively moving. The last phase makes the dot move in a Lissajous-like motion which is more fluid like you are suggesting.

The challenge is recording and syncing the motion at a higher frequency and being able to save without much drift and the performance of these landmark/gaze models is often slow.

One more option to speed it up is not to do the eye tracking at record time, just record a crop video of the face and the screen first at 60Hz and then run the model on each frame and update the metadata of the dataset.

The WebGaze software used in this page was introduced in by Papoutsaki and co-workers by a team from Brown university and Georgia Tech in 2016:

- Papoutsaki, A. et al. (2016). "WebGazer: Scalable Webcam Eye Tracking Using User Interactions." Presented at the International Joint Conference on Artificial Intelligence (IJCAI). https://www.ijcai.org/Proceedings/16/Papers/540.pdf

While we are at it, you may also find the following research publications relevant to this discussion:

- "Improving User Perceived Page Load Times Using Gaze" (USENIX NSDI 2017) https://www.usenix.org/conference/nsdi17/technical-sessions/...

- "No clicks, no problem: using cursor movements to understand and improve search" (Huang, White, Dumais from Microsoft Research, ACM SIG CHI '11) https://dl.acm.org/doi/abs/10.1145/1978942.1979125

- Virtual gazing in video surveillance (SMVC '10) https://dl.acm.org/doi/abs/10.1145/1878083.1878089

Ha! The timing is impeccable. This is a great demo. I've been experimenting with using gaze and eye tracking for cursor prediction as a side project. I like the idea of pressing 'space' for each dot. I just had the 9-dot movement going from one point to another. I'm using Mediapipe's face landmarks model (I wasn't aware of WebGazer). I'll continue to work on it, but it's great to see a similar thought process.

I may be the result of some evolutionary bottleneck, but wherever there is a camera lens, I assume eye tracking (and sentiment prediction) is at least possible, and at most globally always on.

Next time you see a digital ad display, look for the little black circle on top of it. Yes, they do. Feel free to put a sticker over it.

Just to play devil's advocate here... I'm about as extreme a privacy absolutist as you can find. But given that we're all on camera all the time in public spaces, like it or not, I don't consider tracking on digital billboards to be inherently evil. It could be used for evil (like - an authoritarian government tracking who looks at a billboard for an opposition political leader, for instance). But it could just be innocent data gathering. If you ran a plumbing business and paid $10k for a billboard, wouldn't you want to know if it was worth it or not? It's not as if decades of focus groups and hand-wavey feelings about what is or isn't effective advertising didn't already steer us into a society entirely dominated by big loud ads everywhere.

People who make products and sell services need to advertise. They in turn pay taxes. The many layers of parasitism in the advertising world historically relied on conning these people and taking their money in exchange for an unprovable proposition, namely that if you run this ad we tell you to run, right here, your sales will go up - but we'll never be able to actually tell you for sure by how much, or whether it was a good deal for you. From that perspective, more and better viewership data helps undermine the advertising bullshit machine and close the gap between people who run businesses and the people they're trying to sell their services to.

Remember this incident?

Italians deploy fearsome SPY MANNEQUINS to win Fashion Wars (2012) https://www.theregister.com/2012/11/22/bionic_mannequins_are...

According to the media at the time, the mannquins quickly disappeared again after the scandal broke. If you are more cynical, you might question that narrative.

I never heard of this, but it's hilarious.

> People who make products and sell services need to advertise.

Do they though?

No one is entitled to better data. It should be on the advertisers to figure out more privacy preserving ways of getting feedback.

Ubiquitous technical surveillance is, in economics parlance, a negative externality.

We're just going deeper and deeper into attention-grabbing displays; we now even have ANIMATED screens next to the ROADS in the Netherlands. Flashing, showing texts like "WATCH OUT! We have a new product!".

I have epilepsy. It's managed, I mean, I'm allowed to drive again. But what about people who don't have it managed (for some no meds will work)? They're just f'd when they walk down the street and Nike NEEDED to switch the screen every 0.2s?

If I can in _any_ way inhibit their ability to grow, I will.

If I need your product, I'll find it. I'm against any and all ads - I know it's unrealistic, but I'll do everything in my power to lower the amount of ads I see or to be a nuisance to them.

Really don't care about morality in this case.

>we now even have ANIMATED screens next to the ROADS

Heh, only 30 years behind the US.

Of course our rate of people dying at younger ages over here seems to be a lot higher too.

I'm right now in Malaysia and my residential building lifts there are TV playing ads and make many tricks to grab attention:

- sound of phone ringing to keep you out of your zone

- loud annoying music with annoying lyrics

- putting cat in advertisement and meowing

Love this, any experimental HCI project managers inspires me to think differently about computers and tech

Impressive work - I have experimented with Tobii trackers - they are pretty accurate to work with. But accomplishing this with a single camera is definitely something. Would love to follow your work further. Keep going :)

Very cool demo! Once this becomes good enough (now it's very wobbly and requires huge UI) I'd love to be able to read articles and navigate just using the eyes. Feels very natural.

I'm glad it feels natural! I wanted it to feel similar to Apple vision pros where the eye tracking just works. Hopefully in the future, I can just curl up in a blanket and let my eyes do all the scrolling when I'm reading.

This is very cool!! Have you considered making a WebGL game that uses eye tracking for things like aiming? Could be very cool and one of the very few accessible games

Thanks for your interest! An eye tracking game is an interesting concept. My concern is that there is already a lot of overhead used for eye tracking that an entire game on top of that would be too laggy to be playable. Also, the eye tracking software right now doesn't seem precise enough for things like aiming. I could totally see a pure standalone game that utilizes eye tracking.

Hey, I just wanted to say this is one of the coolest demo's I've seen all year. This is really, really fun.

Thanks! It means a lot that people also find my stuff cool

> Screen Too Small This application requires a minimum screen size to function properly.

Why?

All the components needed to be big enough for eye tracking "selection" to consistently work. It's frustrating when you are trying to look at a button to press, but the eye tracking thinks you are looking elsewhere. If the screen were any small, the components wouldn't fit. I believe the dimensions should work on most computer screen sizes and just bar out mobile devices. This was also intended since the eye tracking does not work on mobile.

> the eye tracking does not work on mobile.

Isn't that a function of platform and not screen size?

> All the components needed to be big enough for eye tracking "selection" to consistently work.

Isn't this also a function more of angles and not pixels? A phone close to my face has the same apparent size as a computer screen farther away.

Reply on news.ycombinator.com