Typing on Any Surface: A Deep Learning Method for Keystroke Detection in AR | Hacker News Reader

Typing on Any Surface: A Deep Learning Method for Keystroke Detection in AR

arxiv.org

45 pointsPaulHoule23 comments

Back in around 2008 at the University of Utah I participated in a study someone was doing about detecting touches on any surface using a camera. When you touch a surface, the blood under your fingernails moves, and that movement can be detected with a camera and turned into a usable signal. I think they were even able to get force and direction as well. It was very interesting, but I haven't seen or heard anything about that method since.

I did, and still do, think it's a great idea.

I can't picture how it would be easier to detect blood under fingernails than movement of fingers. Is this some kind of device attched directly to the fingers, rather than detecting collisions with the surface "remotely"?

Motion amplification makes this feasible. Not sure about easy in any case.

http://people.csail.mit.edu/mrub/vidmag/

I think I can see the colour change without any of that when I touch my phone screen, my fingernails look more red when I’m pressing down

Mine turn white/lighter colored. How hard I'm pressing down is easily visible based on the size of the discoloration.

Looking again I think if I push with the middle of my finger then it goes redder but if I push at at angle with the edge it goes whiter

I don't think this will be useful unless that find a way to provide tactile feedback, too. Otherwise, you'll probably have a lot of ghost presses due to fidgeting movements.

I wonder if a very precise haptic (in the HMD, or maybe wrist worn?) response + sound cue would be enough to make it tolerable.

Hopefully something along these lines gets incorporated into visionOS before launch (or as a v2 fast-follow). Typing against a floating keyboard in thin air is not ideal!

All these AR headsets lack tactile feedback

I'm not sure where mine got to, but the Tap Strap worked okay typing against my leg. https://www.tapwithus.com/product/tap-strap-2/ I don't recall whether I had version 1 or 2, and I have no idea how the XR behaves.

As someone who has an uncanny ability to break at least one key (usually the E key) quite quickly on every keyboard he buys... I look forward to being able to type on a slab of concrete.

That won't be healthy for your fingers though.

A rubber mat should work too

Sounds like you need a hall effect based switch keyboard.

Thanks, I will look into that. The biggest problem is finding a solution for laptops, the last three of which I had broken a key within the first few months.

thats actually kinda impressive! Perhaps ive been lucky with them but I dont know if Ive ever had a keyboard key break. I think maybe on some old cheapo laptops I had when I was a kid the backspace would ghost after a few years? but actual keyboard peripherals, no.

Personally this tech sounds like a fun novelty but for myself I rely on the tactile feedback from keys to know when ive made a mistake. I think I would spend a whole lot more time dealing with unnessacary nonsense in programming if I didnt immediately know when I rolled over two keys or didnt quite depress one far enough. It's part of the reason I really like mechanical keyboards at home although the noise is too much for office use.

for good typing you really need buttons. the issue keeps getting danced around, but ultimately I think where you want to end up is where you have a device like a twiddler on one hand that somehow is easily retractable out of the way.

or, you have a pocketable keyboard like the (failed to launch) textblade.

foldable bluetooth keyboards come close but they need to be able to fold smaller.

This sounds like an interesting idea but it is not novel. Also, the solution below has what other comments have had on lack of tactile feedback.

https://www.amazon.com/AGS-Advanced-Projection-Bluetooth-Sma...

I wonder how effective this will be as, AFAIK, most people can't actually touch type.

Then again, I didn't know I could touch type until I did once have to type on a keyboard I couldn't see.

Perhaps this will spur a renaissance in touch typing.

Would really like to see a reasonable way to control an ipad or computer while in the kitchen, hands covered in flour. Let me type and use gestures on the countertop so I don't mess up my device.

91% accuracy. Anyone know what the accuracy of typing on your typical iPhone keyboard is?

I can attest personally that my accuracy while typing on my mobile is abysmal and (assuming it isn't just me) I can believe we're all ok with this. I was much better and faster on my Nokia with T9...

Reply on news.ycombinator.com