Ring-Based Mid-Air Gesture Typing System Using Deep Learning Word Prediction | Hacker News Reader

Ring-Based Mid-Air Gesture Typing System Using Deep Learning Word Prediction

arxiv.org

62 pointsPaulHoule34 comments

As a user of the original Leap Motion Controller, I see one problem with this solution: the dreaded gorilla hand or in other words, gradually increasing numbness and tiredness of the hand when doing mid-air gestures.

Personally I couldn't work longer than half an hour with the device. There's something about needing to position your fingers in a precise location in space that's particularly tiring.

"RingGesture, a ring-based mid-air gesture typing system, enables users to input text both quickly and accurately. The process unfolds as follows: a) The process begins when the user articulates their wrist, positioning the cursor over the initial letter of the desired word. b) Then, the user performs a pinch gesture with their thumb and index finger, marking the start of the cursor’s trajectory. c) Subsequently, the user gestures the word’s trajectory in mid-air to complete the input by articulating their wrist. d) Upon releasing the pinch, the deep-learning word prediction framework, Score Fusion, predicts Top-K words, with the Top-1 word being pre-selected."

Oh. It's just a virtual keyboard in VR/AR with phone-type word completion. The title suggests something more like this scene in Minority Report.[1]

[1] https://www.youtube.com/watch?v=33Raqx9sFbo

The QWERTY keyboard layout was designed in the 1870s in part to make it hard to type so fast that typewriter keys would jam. With text-to-speech and the gesture and movement recognition abilities described in this article, is typing really input method we should be optimizing?

I believe it's a misconception that QWERTY was designed to slow down typing.

> There’s some dispute over how and why Sholes and Glidden arrived at the QWERTY layout. Some historians have argued that it solved a jamming problem by spacing out the most common letters in English; others, particularly more recent historians, hold that it was designed specifically to help telegraphists avoid common errors when transcribing Morse code.

https://www.smithsonianmag.com/innovation/origins-qwerty-key...

> is typing really input method we should be optimizing?

Operating large amount of keys with all your fingers to compose and invoke complex commands is a powerful idea - especially given that people can and do learn to navigate complex systems this way intuitively, in the same way drivers eventually internalize the interface and stop thinking about it.

Having those buttons labeled with letters in a particular layout - that's not where the power of this input method comes from. Perhaps it was a necessary Schelling point, though - QWERTY keyboard is a good default. Without it, computer vendors would be tempted to experiment with their input panels, get creative with controls, making them effectively unique per model.

Apple already screws with people's head remapping how the keys work to psychologically impose a bit of vendor lock-in once they get used to it.

I swear, every time I have to use a Mac and the End key accidentally goes to the end of the document instead of the end of the line, I get a picture in my head of this smug looking guy in a black turtleneck laughing at me.

(And yes, Mac people, I know about Karabiner. Don't bother telling me how I can adapt or how great and superior your Mac is.)

> the End key

Surprised to hear that there is a user of this button. I don't think I've ever used them, and in the last decade or so I've bought keyboards that don't have them at all.

Surprised to hear there are people, presumably in tech, that don't use the end key. I go out of my way to find keyboards that have end and home. Makes it easy to get to the the of the line, or end of the word. And frustrates me to no end when I'm using the mac

And here I’m surprised there are people, supposedly in tech, that don’t just auto hit ctrl-a or ctrl-e (or ^ or $ or A) to get to the start/end of lines. To each their own, I guess.

^a and ^e don't work on Windows and most general programs for getting to the start or end of a line.

You really should say C-a and C-e, if you're referring to the emacs commands. For the vast majority of modern computer programs, ctrl+a is "Select All".

The windows experience for string manipulation out of the box is 10000000% better than macOS.

There might be some configuration or third party software I can use to emulate it, but on windows I can treat space delimited words and lines with much more granularity than I can on macOS

I'm gonna have to massively disagree with you there. MacOS has (basic) emacs motion keys in very nearly every input in your entire machine, regardless of what app it is or who built it or what purpose the input is serving. And cmd-[common stuff] is frequently handled at the OS level via menus, so apps roughly never screw it up, and you can customize it across all of them.

Windows and Linux have absolutely nothing on OSX's consistency.

>has (basic) emacs motion keys

well there you go, youve lost the majority of people who use computers. If you use emacs and/or vim or their various flavors then you are the equivalent of an f1 driver questioning why anyone uses an automatic vehicle.

Id posit the fact that in my day jobs where I am issued a mac laptop that then plugs into a kvm switch where the keyboard does not match the same key commands as the OS unless I set up a ._ file, and that this has happened at multiple jobs going back to pre covid era, as a sign that macOS is actually a worse user experience.

The customer is always right after all

C-a, C-e, and some other Emacs basics shared by readline are universal defaults on, for instance, macOS. They don't get in anyone's way there.

I only buy keyboards, even when I wanted a compact one, with home/end keys because it's so much more productive especially in coding in a terminal.

How do you move the cursor to the end of the current line of text you're editing?

Command+Left or Right arrow

On a mac, ⌘ →

$ or Ctrl-E.

...do you use Macs? It's a lot less common to want to quickly reach the end of the document than the end of the line.

Ctrl-end to end of document, used by most other OS, is a lot more sane. There was absolutely NO technical reason to change this behavior, other than to stockholm Apple users who might think of leaving the ecosystem.

Talking all day, and/or waving your arms around in empty space, is absolutely exhausting and a regression from typing. This paper is about people who are walking around AFK, wearing AR glasses.

edit: ok, waving your arms around all day might be good for the delts and lats, but shoulder RSI is no better than hand RSI.

I dunno.

I find speaking to computers (say Alexa, Siri, Cortana, …) to be tiring compared to speaking with people and slower than typing.

Typing in VR with visual tracking or controllers is practical if you need to log into something or bash out a Tweet, but you want to get a real keyboard aligned with your space if you are going to type much.

The myoelectric device that was used with Meta’s Orion demo could lead to something better, you can get signals from a EEG or my electric array and train it and train the human to communicate in symbols across it. With a direct brain interface, do any better.

Some of my distinct style as an HN commenter comes from my perpetual fight with my iPad. I am sitting in the couch holding it in landscape mode with two hands and typing with just two thumbs. I can do it crazy fast. The text input system has a predictive model which catches and fixes many mistakes that I make (don’t want to turn it off) but also injects errors of its own. (It just did it.). I don’t catch all of these so you find grammatically probable but semantically wrong errors all over what I write.

I'd love some chording gloves or even just something that senses finger motion from say forearm muscle activity. I'd use it with my current laptops - just lean back and twitch your fingers

Agreed. Pair it with glasses (like the Orion) as a connected monitor and you could hack away at a side project anywhere and everywhere.

I actually tend to do a lot of coding in bed and would love this kind of setup. Or while on an airplane where space is a premium (I hate having my laptop on that stupid tray table).

Pair the glasses with bone conductive headphones and you could be immersed in your world without the silliness of the Apple Vision. And you wouldn’t have to turn anything off during takeoffs or landings…

agreed! that's apparently the (eventual) goal of the wristband Meta is using with the Orion headset.

if it could be refined to the point where it'd be as capable as smartphone thumb typing and deployed as a standalone input device would be a real game changer.

I find text with grammatically probable but semantically wrong errors much harder to read than text with spelling or grammatical mistakes.

Basically, it makes the errors harder to spot and the search space to find what was meant larger.

There is sometimes value in a private or silent channel. If public use becomes more common, I definitely hope not to hear people dictating to their devices.

Text-to-speech is hard to do if you are in a phone call at the same time

I thought part of it was so salesmen could quickly type the word:

  typewriter

all characters of which were on the top row of keys.

That doesn't sound true. The quickest way would be for the keys to be in the home row.

After it purchased the device, Remington made several adjustments—including switching the period and "R" keys, so that salesmen could impress customers by typing "TYPEWRITER" using the keys in the top row—which created a keyboard with what is essentially the modern QWERTY layout.

https://en.wikipedia.org/wiki/Sholes_and_Glidden_typewriter#...

And on the full-fledged QWERTY layout article, this claim is called "not formally substantiated". The actual citation in your link is from a collection of essays on science, paleontology, and evolutionary theory - exactly the kind of place that such an apocryphal claim could have been repeated for the sake of a pithy anecdote, rather than a reliable source on the history of typewriter layouts.

Wikipedia editors' tendency to duplicate information in sub-section summaries is a great pain, due largely to this problem - who keeps the editorial voice, references, etc. consistent across all these non-linked articles?

I want to see some magic of AI in the medical world..

Reply on news.ycombinator.com