Handsfree.js – integrate face, hand, and/or pose tracking to front end projects | Hacker News Reader

Handsfree.js – integrate face, hand, and/or pose tracking to front end projects

github.com

187 pointsdetaro35 comments

I'll use this post / opportunity to ask the HN community

(I understand that this is a technically complicated, and potentially sensitive subject)

I've been approached by a few people who are doing what they can to lower the cost of prescription glasses. It's a mix of (licensed) ophthalmologists, people interested in offering community health services, etc.

What is the current state of the art for pupillary distance (PD) measurement based on face detection / pupil tracking using laptop webcams or mobile (front) cameras?

I imagine (obviously?) that there must be scholar research on the acceptable error margins for a PD measurement (depending on the type of vision condition, i.e. farsightedness, etc.)?

Would using something like Handsfree.js or https://github.com/esimov/pigo (which has pupil detection) be a good start, or would these be an ~order of magnitude off in terms of the necessary margins?

Thanks a lot.

It comes down to physics. Cameras and their lenses come in all shapes and sizes, most of webcams don't expose their geometry in a programmatically accessible way, even if they did, most of them don's have means of determining distance to the subject.

We can detect faces, eyes and pupils, but with no reference points we have no idea how big or far apart they are.

Thanks!

What about sites (such as Warby Parker) that make you hold up a credit card (which has a standard size) and use that to determine (via what I assume is a rule of three) the PD? [0]

I can see of course problems with the card not being “perpendicular” and other 3d distortions.

[0] - https://www.warbyparker.com/pd/instructions

Hello! Handsfree.js doesn't yet have pupillary distance, but I do plan to implement MediaPipe Iris very soon which does: https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-i...

This probably isn't very helpful, but I'd imagine underestimating the PD is going to lead to happier patients even if not all that accurate. We can accommodate with crossing our eyes, but the opposite is not usually true.

Fwiw, there's https://www.eyeque.com/pdcheck/ .

And Google's MediaPipe Iris used iris diameter for distance estimation.[1]

EyeQue is perhaps a cautionary tale. "Change the world" "few dollars of optics + phone = D+CYL" become $45 one-person-per-subscription "vision tracker".

[1] https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-i...

If my memory serves me correctly, I once a used a digital try-on app from an online glasses retailer and they had me hold a common item with a known width (something like a credit card or drivers license) in front of the cam within a bounding box. It could guess the location of my pupils but allowed me to drag the pupil location markers to the exact centers. Calculating pupillary distance was then just a matter of using the distance in pixels between the pupils and converting them to mm using the ratio of pixels to mm of the card in the previous photo.

There are a lot of problems with measuring pd accurately with a photo. Ideally you need to ensure the pupils are aligned (i.e. the head can't be turned) which is really hard to tell from a photo (there's no depth).

The credit card trick somewhat works since you can guage depth given the size and positioning of the card, but the errors still can end up in the 2-3mm range fairly easily.

The best mechanism would be to create something like a Google cardboard to guarantee the head is properly positioned and then perform the measurements.

There's a nice demo of using handsfree.js to drive a car in a racing game -- https://twitter.com/GoingHandsfree/status/140032233301379891...

He just posted another demo where you kick your feet to accelerate a la The Flintstones -- https://twitter.com/GoingHandsfree/status/140136648492059033...

Pretty fun to watch!

His comment: The controls are whacky as hell and it took 5min of practice but it's just a tech demo

...

haha yeah, the idea is to do one radically different use case every single day for 100 days straight so I don't always have time to make them smoother

I did do another whacky one today that is a lot easier to do. It's called "Flintstone Mode": https://twitter.com/GoingHandsfree/status/140136648492059033...

Wow this is amazing! I remember a few years ago in my thesis I was trying to use something similar in ROS to detect hand gestures and make a robot drive toward you, it's incredible to see this in the browser!

Oh that would be so epic! I did something similar during my residency for Handsfree.js but for "puppeteering" a robot: https://giphy.com/gifs/1XE2rnMPk6BFu8VQRr

On Safari for Mac, you must enable Develop → Experimental Features → WebGL 2.0.

I have tried the underlying library, MediaPipe/TensorFlowJS, and it is only okay. Once you try to use it for visual augmentation, I found it doesn't track that well and it has problems with scale.

So, while it may look good with the skeletons, notice they do not have useful demos, mostly just proof of concept demos that do almost nothing.

I'm very sorry you don't think Handsfree.js has useful demos. That kind of hurts because I put a great deal of effort into making them useful as starting points (I started all this while I was homeless to help someone). I work on Handsfree.js full time and my goal is really to create simple demos so that others like yourself can run with it!

Maybe you'll find some of these more useful?

- Face Pointer, and accessibility tool to help people who can't use a mouse/keyboard: https://handsfree.js.org/ref/plugin/facePointer.html

- Palm Pointers, scroll the page and even multiple scroll areas at once: https://handsfree.js.org/ref/plugin/palmPointers.html

- Gesture Mapper, map static poses in seconds: https://handsfree.dev/tools/gesture-mapper/

- Face Coding (not ready), to help people with disabilities code by snapping blocks together: https://twitter.com/GoingHandsfree/status/140056015231972966...

- The library itself doubles as a Chrome Extension, here are some examples of what you can do with it: https://handsfree.dev/sites/

I have dozens and dozens of more experiments. Most of them are pretty basic, but the point is to show you what's possible :)

Also, here is a simple demo that has been helping some of my friends with disabilities. It uses the Face Pointer to play desktop games: https://twitter.com/GoingHandsfree/status/140022171083874713...

Have you tried the newest MoveNet from google? That thing works really well. I only started experimenting with the Lighting model (not Thunder) for our VR game but even that is pretty robust.

If you want to see how stable it is, have a look at that: https://youtu.be/zz9S5hgrWpM?t=147

It's tracking me with the Oculus Quest on the head, so it does not even have facial features to go on

Thanks for the link - Does MoveNet give you 3D positions / rotations for each joint or just a 2D position on the image?

Just the 2D positions plus a confidence level. In the video we are using only one view so it's no true full body tracking yet. We are currently trying to use two views to get a rough 3D tracking. The good thing is that the positions of the detected points are relatively stable so with a big enough baseline that should work out for a rough 3D pose estimation

BlazePose does give 3D portions for non-facial key points

Very cool project! This looks like so much fun to build. Thank you for the great documentation of your endeavour!

What are some of the applications for this?

I can't speak about this specific project but pose tracking in the browser can actually be really useful to get it on mobile without the need for dedicated apps.

We are currently experimenting with using phones (or tablets or basically anything with a browser and a camera) as a tracking source for full body tracking on the Oculus Quest ([1]) and being able to just use a Tensorflow model in the browser has cut down the time of development on the mobile device side to almost nothing.

There are of course a whole lot of other issues one buys into with that approach (browser security policies, the whole thing not working in certain countries due to problems with loading the model) but that's manageable

[1] https://www.youtube.com/watch?v=zz9S5hgrWpM (relevant part starts at 0:29)

I found this demo of him playing "Into the Breach" with only Face Gestures very impressive! https://twitter.com/GoingHandsfree/status/140022171083874713...

I made a slouching detector a while back. It calibrated the distance of your face from your laptop and if you start leaning in it’ll give you a notification to sit back. It’s a bit dated but still works on Chrome and FF, I believe:

https://slouchcam.com

https://github.com/midiblocks/handsfree#examples

You could integrate this with adtech to help make advertisements more interactive and to analyze how the viewer is receiving the ad by their pose and facial expressions. Based on that you could have an algorithm serve only ads that improve facial expressions and poses to positive ones

I hope we haven't yet gotten to the point where people will mindlessly click "Yes!" when a random website asks for webcam access.

Another excellent reason to disable JavaScript in your browser

Could play punch the monkey!

WebXR comes to mind.

Probably the coolest library I've seen in a while.

the library and the demos are not loading for me.

Reply on news.ycombinator.com