Comment by mncharity | Hacker News Reader

Comment by mncharity | Hacker News Reader

Parent Full thread

Has anyone seen keyboards with non-traditional touch surfaces? Such as trackpad keycaps?

Backstory: My fingers slide smoothly across my ThinkPad laptop keyboard, so I was playing with optical hand tracking (with a surface mirror to better track touch) to make the entire keyboard (and surrounding laptop) into a multitouch surface. There were occlusion and jitter issues, even with fingernail markers, and I set it aside while still not quite working. But I was left thinking "I should get back to this" rather than "never again". Stroking a keycap as a trackpad seemed nice, even already having a touchpoint. Merely touching modifier keys (emacs) rather than having to press them. With a 3D display, I overlaid video of the keyboard, slightly above the screen plane so it was easily seen but only moderately annoying. And thus could overlay a multitouch control panel that doesn't require shifting attention from the screen, or hands from the keyboard. With screen-comparable VR/AR seemingly only a year or three away now, with attendant changes in UI constraints, there seems an opportunity to escape decades-old HID fetters, at least with professional UIs.

Yeah, Apple got their multitouch technology in 2005 by buying Fingerworks, the startup that made Touchstream keyboards, which were sort of like a giant single trackpad with key areas on it. Now that every Android phone and tablet has this feature, you could maybe use a tablet or two as a reconfigurable keyboard.

Myself, I really want more touch feedback from my keys, not less, but to each her own!

And there's currently the Lenovo Yoga Book C930 2-in-1 with an e-ink keyboard surface.

But as you say, not a high-end keyboard experience.

The optical hand tracking I mentioned was cameras on sticks perched on an existing laptop. So the multitouch was in addition to, not a replacement for, a niceish-for-a-laptop thinkpad keyboard. Well, except for modifier keys without the pinky exercise of actually pressing them (which may or may not be actually viable). If you ever say tactilely run your hands over your keyboard, and say drum your fingers, but all without sufficient pressure to press any key, it was that experience, added to the usual keyboard experience. With 3Dish hand pose and motion available, the two modalities might demultiplex fairly cleanly.

Even for programming in VR, I'd like to avoid UX regressions, and so start with a nice keyboard. Though I was happily surprised by being able to hold a chopstick without it much degrading typing. Providing a nice in-hand high-precision 6DOF controller. And potential 6DOF (pressure) stylus - ball on the end let it sort-of glide over keys... but jitter and pragmatics would need improvement to be viable.

I don't think OP is referring to a touch keyboard, but rather that the keycaps of each key would be one part of a larger multi-touch surface. As your finger left one key you could sort of "guess" the direction it was going and have something close to one big multi-touch panel that was composed of smaller surfaces. But I could be wrong as well.

Indeed. And with a keyboard like this[1], the gaps between keys are smaller than (my) finger width, so a sliding finger might provide continuous data.

Optical position tracking of fingers was actually ok (but not on a bus, and not so much for stylus), and with a look-along-the-keyboard-surface mirror, even optical height and touch was ok. But the touch wasn't great, and occlusion was a "keep it always in mind" problem. So I would have been delighted to find say a low-precision multitouch textile, to hopefully drape over the keys without too-much impairing typing. To decouple a fast low-precision-XY high-precision-Z touch, from simple and adequate camera XY. But crafting a capacitive grid... seemed a yak too far. All this was yak shaving - I actually wanted to be exploring VR IDEs catalyzing improved programming languages. :/

[1] https://upload.wikimedia.org/wikipedia/commons/c/ca/ThinkPad...

I had this idea a while back as well. What brought it to mind was wishing I didn't have to take my hands away from the keyboard to mouse around.

Anyway, Apple has patents covering this idea. I believe 9,041,652 is the one I'm thinking of.

Yes, patents have been amazingly toxic for HID progress.

Commercial HID innovation seems to be bimodal, DIY and bigco, with a patent-induced desert in between. Take multitouch. Years of crippled small companies, selling kits to dodge patents, to small DIY communities (DIY historically being much less of a thing than now). Speech-to-text and Nuance. Hand pose from webcam and colored glove research, becomes of-course-unsuccessful startup, bought by bigco, and thereby unavailable to support further innovation. Leap Motion's DIY Project North Star is an ongoing example - there's software, and you can buy parts, but not devices. So while there's an active small-company ecosystem around say keyboards, that falls off really fast. Small-scale markets of oem support for exploration and innovation... aren't an interesting niche for bigcos. So they produce patents and not products, and even proven too-small-just-a-distraction-for-us markets are abandoned.

As education innovation starts overlapping ML and tech, it looks to be picking up similar dysfunction. Ah well.

The Blackberry Passport had a keyboard like that. You could glide over the keyboard to scroll and other gestures. I don't know if the new breed of Blackberries have it, but it was a great feature.

The Key One and Key Two have it too.

The Waytools text blade sounds somewhat similar to what you're describing:

https://waytools.com/

Interesting. Thanks. So an ortho keyboard with trackpad keycaps that span multiple keys. I wonder where they sourced the one-key-wide pads? This might be a relatively painless way to introduce trackpad keycaps into an existing keyboard ecosystem.

Product-wise, they seem to be in a multi-year struggle to get to production.

Web-dev lesson: A video-based ui, emphasizing closeups, where the video visuals stall, but music continues, with a hidden-by-default slider and no spinner, is an unfortunate combination.

Yes. https://vimeo.com/258421095 It's possible to turn a single keycap into a touch surface with the mechanics of a TrackPoint, and mapping to VR. Look for a product this year.

Oh, nifty. I didn't find a page to track keyboard status, but here's http://www.tactuallabs.com/ .

A depth camera can give rough finger height[1], and multi-camera care and kludgery can do even better, but, that last little bit, of high-speed high-reliability millimeter-ish is-it-touching-the-key-yet/still-or-not contact determination, is something I've not seen well addressed.

Given this unicorn-like keyboard, I wistfully note that having pressure sensing would be very nice too...

FWIW, I also note one way a company can easily support linux, often overlooked, is to provide only an as-is unsupported binary-blob library with any secret sauce, and leave it to the community to link it, wire it to IO systems, and deal with OS variants. The payoff, for a product throttled in part by community innovation and/or adoption bootstrap, is that while linux is often a small market, it's a highly-disproportionately innovative one. For this keyboard, the envisioned market may be mostly gaming, and thus mostly Windows. But it might be a youtube "look at my neat hack" video, which prompts some game dev to support it.

[1] https://frl.nyu.edu/multi-touch-with-realsense-camera/

Yeah, this does that last millimeter in Z. It can do pressure, at least by virtue of knowing the force needed to move the keycaps and how far they've been pressed.

An SDK has to be part of this. I agree that Linux support will be good to have, especially if there are people like you to play with it. We should talk.

Previous thread on trackpoints:

https://news.ycombinator.com/item?id=18811314

Oops, yes, s/touchpoint/TrackPoint/.

For those who have previously tried them, and been unimpressed, I note that driver configuration can make a _big_ difference. A new-linux-driver and old-firmware issue has me now at merely "it's ok", but switching to an older setup was an "OMG - how could I have forgotten this?!" experience.

Reply on news.ycombinator.com