Hand-Tracking with Three.js
rdtr01.xl.digital
rdtr01.xl.digital
Glad to see the conversation about hand tracking in the browser over here.
This demos is done under the context of a series of creative experiments on how to use real time hand tracking in the browser for creative interactions. Will be posting more experiments soon.
Tech background: I am using MediaPipe to control the hand rig in threejs. MediaPipe provides landmarks that are used to control a threejs Skeleton (hierarchy of bones with rotations).
Feel free to ask, I will answer any questions!
Fwiw, some things I've found fun: Clip-on fish-eye lens, intended for phone but fitting on laptop, for expanding webcam field of view. Additional cameras: on sticks above screen tips for high-res stereo positioning over kbd; asymmetric high-off-to-side to trade some resolution for some field of view (meh); high-overhead for whole-workspace tracking. Binocular periscope with webcam splitter and screen-tip mirrors (blech - low-res awkward fiddly). Look-down mirror on webcam, partial or full, to get kbd view (nice in VR). Look-down with curved mirror along top of keyboard to get "out along kbd surface view" and crufty touch detection for kbd-as-touch-surface (cute but fiddly - only makes sense to save a camera or two; caveat I had high-contrast white hands on black thinkpad kbd). Putting tracking markers on fingers (flats, a-frames, or cubes on velcro rings) makes for less jittery tracking, but is awkward (meh). Markers taped around keyboard help with calibration.
Magic wand. I found I could more-or-less manage to type while holding a chopstick. So stuck a marker cube on one end, and an arc-sliced-off a small-Xmass-ball on tip, so it slides smoothly across (thinkpad) keys. Barber-pole rotation marker. Anvil'ed tip pressure sensor, a finger microswitch, and very thin and soft ribbon cable to arduino. But didn't actually get the pressure sensor working before punted on all this. Chopstick was narrow enough to avoid breaking hand tracking.
Some gotchas: 2K camera resolution was painful for tracking. (Several years ago) mediapipe finger tracking was annoyingly noisy for doing stereo. You only get one usb2 camera per usb port, even if it's usb3 (maybe usb3 cameras allow working around that limit nowadays?). If you do hand, arm, face and marker tracking on several cameras, even with native gpu mediapipe, you're burning a lot of gpu just on the human interface device, before your likely-graphical-itself app even starts. If I had it to do over now, I'd punt mirrors, use 4K usb3 cameras, and at least with desktop, more cameras. Nicely merging high-latency camera tracking with lower-latency keyboard, touchpad, and graphics tablets, requires changes to the input event pipeline, and adapting apps to deal with "oh my! That space key pressed several keys ago - it was pressed with a pointer finger at position 3!, so that means we roll back app state and then ...".
Here we are a half-century later, still banging on glorified xerox altos. We're so broken.
Greenfield, stylus-wise, I'd... (1) Punt "keyboard as graphics tablet" as a bodge. Except for Mac-ish non-tiny touchpad with stylusified 2: (2) Simple chopstick with color ends and barber pole. Caveat that color blobs with 2K cameras and ambient light are noisy low-res (and slow) for small gestures. Workday ergonomics says resting hand with small motions, rather than movie arm waving. Barber pole pushes towards full-not-pen-short chopstick. And I've not seen a nice simple story for finger pressure/buttons. Sensor fusion with hand pose might be interesting? Squishy HID? (3) Graphics tablets already give clean fast high-res 2D, sometimes several cm above surface, often pressure, sometimes tilt, even rotation. Might add high-latency optical height. Gestures with distinguishable 2D projection could skip burden of high-time-res fusion, for a fast dev path to UI software rather than HID struggles. Caveat mediapipe hand pose struggles with thick black stylus, black background, and white hands (at least years ago - maybe someone is now training with styluses? Maybe someone's stylus, if recolored, is thin enough?). For the other hand, can fuse pose with tablet multitouch. Fwiw.
I do that poorly, repeatedly. A mindset of "today's rev n is bad, still unusable; tomorrow's incremental rev n+1 will be slightly better; no point in recording bad, wait for better; will demo at meetup for friends, but otherwise, who'd care?"... left a sparse trail. Sort of: you might take a picture of your nice finished cake, but of baking? There have been HN posts of commitment-hacking as a service, eg, iirc, a Japanese workspace with sign-in like "I'm here to write one chapter, and I'd like person-standing-behind-me level pressure". So perhaps, motivate documenting this week's state as a service? As finding/creating community that's interested in such seems often difficult.
Hmm, here's a snapshot[1] of my late-rev laptop hardware with flop-up kbd cam and (stowed fold-up) stereo cams (wires not connected). Gaff tape, sticks, vecro and cardboard esthetic allows fast and incremental iteration. For wires, I like magnetic usb connectors[2]. Fwiw.
[1] https://twitter.com/mncharity/status/1232446953784369154/pho... https://pbs.twimg.com/media/ERqCfdkX0AEWTN_?format=jpg&name=... [2] https://twitter.com/mncharity/status/1255300177960808448
As a matter of fact, lots of people over twitter have been sharing their frustration when they attempted to do the same thing.
This is the funniest one:
https://twitter.com/isjackwild/status/1617559339891396619
Here's some comments on my implementation:
https://twitter.com/SketchpunkLabs/status/161758661970323049...
[1] https://www.searchenginewatch.com/2013/10/03/google-acquires...
unfortunately the rendering of the hands in the large window jumps all over the place on firefox, ubuntu, razer laptop. :-(
That said, I still think MediaPipe is an excellent piece of ML tech. And, from a dev point of view, quite easy to get working for various things in the browser[1][2]
[1] - MediaPipe Selfie Segmentation for real-life background replacement - https://scrawl-v8.rikweb.org.uk/demo/mediapipe-001.html
[2] - MediaPipe Face Mesh for, well, drawing lines on your face - https://scrawl-v8.rikweb.org.uk/demo/mediapipe-003.html
There is some jumpiness but I am holding the phone in one hand and making hand positions with my other so I'm less concerned about the jiggles.
It has a hard time naturally when all of my fingers are occluded by one another.
Way to go.
Not quite high enough fidelity to handle ASL though.
Some issues I ran into testing it, if it's something that interests you:
- Cannot distinguish closed vs. open fingers (always adds gaps between fingers, even if they're touching) (B) - Can't handle crossed fingers (R) - Doesn't seem to like extended vs. curled fingers in some cases (H) - Other failed letters: (Q), (E?), (F?), (G), (Q), (S), (U/V)
But, when signing naturally, it seems to get enough of the shapes and orientation correct enough to understand what I'm seeing. I'm sure there's things it'd trip up on because of some of the above weaknesses in detecting hand shapes but it does seem to get movement, orientation, and position "good enough".
An "air-piano" seems well within the realms of possibility now.