Control Linux based distros using hand gestures using OpenCV, GTK, Mediapipe | Hacker News Reader

Control Linux based distros using hand gestures using OpenCV, GTK, Mediapipe

github.com

79 pointspolitelemon19 comments

In the screenshot they’ve mapped fist to “shutdown now” which seems to either indicate high confidence in the gesture recognition capabilities, or a good sense of humor about the whole thing.

Some people want to live life on the edge. Expert mode would be a gesture that nukes your installation

Suggestion: middle finger mapped to "run last command with sudo".

I would prefer looking up towards the ceiling combined with a deep breath, followed by muttering a four-letter word.

Are you by chance interested in a command named after the four-letter word, which, when invoked, automatically fixes typos in the last run command and reruns it: https://github.com/nvbn/thefuck

For those who don't know, this works in many shells:

    sudo !!

Starting in the early aughts and extending into the release of the Wiimote there were a handful of projects like this, largely for Mac. Never amounted to much more than a proof of concept most of the time but I always found the concept exciting. I had a lot of hope for Magic Leaps initial stand-alone hand tracking device before they pivoted to AR

At the end of the day, for my computers, my favorite hand gestures all involve the keyboard.

Yeah I keep a webcam pointed at my keyboard too. That way the computer knows I’m subtly flipping it off when I undo in vim.

I love this, I can't tell if it's really true or not

I feel like a few gestures to lock or switch workspaces would be really ideal.

I worked on something very similar to this a couple years ago. My main issue with it why I didn't go beyond the PoC stage was that raising my hands up to the camera soon became a pain (I use a laptop), esp as I have it at eye level. It also had a hard time keeping up with my motions, so most of the time it was getting a blur.

I had a overhead-ish camera on a stick, that flopped up from the laptop screen back, to point straight down at the keyboard. Alternately, a phone clip-on fish-eye lens was low-res but ok, iirc. Kludgy little mirrors were iffyer, but there's a commercial one for iPad I didn't try.

Blur wise, perhaps it was 30 fps? I was mostly ok with 60 fps. Would have liked 4k resolution for markers and precision though.

More problematic was integrating high-latency "wait for the camera frame and then for mediapipe" with low-latency "fast keypress response". Drove me towards replayable event streams, and thus custom apps. But I was using it for custom XR-ish apps anyway.

Running high-res mediapipe etal on multiple cameras kept a thinkpad doorstop laptop warm even before starting XR. Keyboard feel got a bit toasted. But TFA doing gestures-only fixed-poses low-res single-camera is far lighter.

It's a pretty old system; had it over 10 years now, so I didn't expect too much re the blur. I'll likely revisit after I've upgraded to something more modern. Maybe by then I'll have the camera position figured as well.

Love to get into the XR space myself.

What is, exactly, a "smile hand gesture"?

By comparing the list to the icons, it would be what I would call a "Finger Gun[1]"

1: https://en.wikipedia.org/wiki/Finger_gun

similar but different, I've used MediaPipe to create Eyeroll, a WebExtension that lets you scroll up and down webpages by blinking green your eyes.

https://github.com/bjesus/eyeroll/

What? Not using LLMs or an expensive cloud based API?

Quite pleasant, eh? I hope we see more projects without them.

> This project utilizes the hand gesture recognition model provided by the Google MediaPipe project.

The rest is just some glue.

Reply on news.ycombinator.com