Comment by panic | Hacker News Reader

Comment by panic | Hacker News Reader

The linked article by Bret Victor on the future of computer interfaces is worth reading: http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...

Of course, Pictures Under Glass interfaces have advantages too. Touching a screen has become a familiar way to interact with technology: a more tactile interface might be harder to learn. To make a screen-based interface, you only need to draw a 2D picture and respond to 2D touches, not design a 3D object which can be manipulated in space. You can buy touch panels off the shelf instead of having to manufacture expensive custom 3D parts.

But these are advantages of easiness and cheapness, not quality. If designers and engineers can put in the time to make a physical interface and users can put in the time to learn it, you can improve quite a bit on Pictures Under Glass. Think about how terrible it would be to replace a car's steering wheel with a touch screen, for example.

So that's really what the article is saying: we should think of touch screens as the cheap, easy option, not the exciting, futuristic one. Our futuristic starship control panel should be a custom-manufactured physical interface instead of a cheap glass panel.

Well, so the thing that I think people don't realize about gestural interfaces is that gestures are hard to do and tiring to do all day. They are also very low bandwidth. Typing is hard to beat in terms of bandwidth / strain. But there are things that gesturing is better at, and I like that that article focuses on that. I think the mistake is to try to get a catchall. That's why we have both a keyboard and a touchpad / mouse. Each does a different kind of task. Touchscreen typing, on the other hand, is a step back. Game console typing is like 20 steps back.

I'm not sure mouse and keyboard will be unseated from the top spot until we get to the point of straight mind-machine interfaces.

Speech to text has been making a strong showing recently. It's been creeping more and more in my life as a faster and easier alternative to keyboard input. It does take some practice, though, so I encourage people not to dismiss it based on their first impressions.

If your Eminem maybe, I can type faster than I can talk.

Siri cuts me off mid sentence because I talk too slow.

Average conversational speed is 110 to 150 WPM. Maybe you type unusually fast? Professional typists might type at 80 WPM and it takes experienced stenographers to keep up with normal conversations.

Again, speaking for speech to text takes some practice, so it doesn't quite make sense to compare a lifetime of typing against your first shot at speech recognition.

Stenographers can type 300+ wpm on phonetic chording keyboards with appropriate predefined shortcuts.

People can also write shorthand with a pen at 300+ wpm.

A big part of the problem with regular typing is that our phonetic alphabet is not optimized for writing efficiency. Sometimes a single syllable takes 5+ discrete characters to type.

Standard QWERTY-style keyboards are also designed to use single finger motions in serial fashion, and requires various awkward combinations of finger motions and plenty of repositioning between keypresses; the design is designed for learnability rather than efficiency, and wastes much of the possible bandwidth of the hands.

If someone with deep knowledge of human hand physiology / neuroanatomy were to design a keyboard optimized for efficiency (at the expense of requiring years to properly master), we could probably push peak performance to 400–500 wpm.

At some point thinking speed is the main bottleneck though. Regular QWERTY keyboards are fast enough to keep up well enough with most writing tasks.

> If someone with deep knowledge of human hand physiology / neuroanatomy were to design a keyboard optimized for efficiency (at the expense of requiring years to properly master)

Per language.

If you're going to talk about stenographers (experts at typing fast, with specialized equipment and training) let's talk about auctioneers.

The world record for speaking is about 600 WPM. The world record for stenographers is about 360 WPM.

A typical speed for speech is 150 WPM, for someone who is practiced but by no means an expert. A typical (but somewhat fast) speed for typing is 80 WPM, for someone who is practiced but not an expert.

So if you're going to compare the fastest speeds or the typical speeds, speaking comes out ahead either way. It's almost as if our ancestors had evolutionary pressure to develop speech, but lacked the same kind of evolutionary pressure for typing.

Regular people (or computers) listening are going to have a heck of a time understanding world-record speaking speed https://www.youtube.com/watch?v=l-o9vTk8Poo

As for auctioneers, most of what they say is repetition and filler. The fast talking is a gimmicky technique aimed at improving sales, not a practical tool for imparting information at maximum possible speed.

As I said before, the keyboard is not remotely optimized for the capabilities of human hands, and English orthography is not remotely optimized for information density as an encoding the spoken language.

If someone speaking had to say a full syllable for every letter, speaking speed would be much slower.

> almost as if our ancestors had evolutionary pressure to develop speech, but lacked the same kind of evolutionary pressure for typing.

More precisely, speech was developed over millennia by millions of people mutating it slowly over time, whereas modern typing hardware was designed by a single person, based on the mechanical characteristics of a particular prototype typewriter in the late 19th century (before any concept of “touch typing” even existed, and typists were forced to adapt to this mostly fixed hardware design as best they could.

So, the fastest rapper seems to talk at 280 wpm and that's not filler.

http://fivethirtyeight.com/datalab/the-fastest-rapper-in-the...

150 wpm speaking is sort of irrelevant when half or more is verbal filler and rambling that doesn't provide any signal.

Go read out a hello world in C++, pressing ; is much faster and more accurate for me than verbalizing "Semicolon". What about variable names? You have to spell out the word each time to get camel case. u-s-e-r-Capitol C-o-u-n-t and that's if it works perfectly.

Python, you have to say return space space space space every damn line. Return space space space space space space space space while inside a function. It would nearly drive a man to switch to tabs.

Navigation, "go 2 characters to the right" isn't pleasant.

To top it all of picture the fun you can have in vim using speech to text. Dear lord.

Now imagine an open office full of people all doing this at the same time. That my friend, is hell.

what about a programming language optimised for this purpose?

This is what I wanted to say as well. To shift to verbal programming, we'll also have to change to a language designed with that in mind.

Interesting convergence would be VR mixed with TTS processing. So, you use VR and presence/hand tracking to physically interact and words to refine and/or program the visual blocks.

Again, same trap. Talking all day is NOT appealing, and doesn't work in a public setting. Works great for mobile phones precisely because often we have our phones out but can't type, like in a car or walking with my hands full.

The other issue with speech to text is discoverability both for commands and the object of the command. This presumably becomes less of an issue as natural language processing gets better. Voice recognition is pretty good with the right microphones today but using it is something like remembering the right wizard's incantation. Think a command line :-)

That's fine if it's a variation on some frequently used command but I find myself forgetting the triggers and skills I have set on Amazon or the names of playlists I might want it to play.

This is actually a great point - I think this is a one-way vs two-way communication issue. With a lot of our current input devices, tactile feedback helps with verifying that the individual "packets" of information are registering, but visual feedback is key for knowing that the high-level command worked / didn't work, and what other options are discoverable. And the fact that the visual feedback is extremely high bandwidth is key. Imagine being read everything on the screen. Come to think of it, screenreaders are hard.

What are some good examples of gestures that work better than alternative input devices? It seems to me like the obvious choices of zooming, scrolling, and rotating things can all be done better with buttons, a mouse, a wheel, or a knob. I can press the back button on my mouse a lot faster than I can two-finger-swipe to go back a page.

I find zooming to be infinitely better done as a gesture - whether on screen or on a trackpad.

The problem is when are you zooming and when are you rotating? Google Maps on the phone is a great example. I find myself using "tap then flick up or down" or just the +- buttons much more useful because they don't require my second hand and won't accidentally rotate the map.

Call me unimaginative, but I think fingers are good at manipulation more so than hand signs. So I can imagine either stuff like 3d modelling tasks, or when you're interacting with a 3d world (think games). Smush-deforming play doh with your hand is so much quicker and more precise than trying to do the same with a mouse or shudder keyboard. Or imagine doing knitting.

I think hands are great when there is a massive multitude of possible next actions, like the knitting example, or artistic applications. Otherwise, if we can logically eliminate options of what the user could be doing (this object can only go in so many slots, etc), it's easy to just assist the user with that (think snap-to-grid) and then a precision input device is less necessary.

> I can press the back button on my mouse a lot faster than I can two-finger-swipe to go back a page.

Big assumption that you're using a mouse (especially considering the ubiquitousness of laptops). Laptops provide a ready-made surface for fully portable use and I've found getting using a touchpad for professional and personal use[0] to be more than sufficient and that the availability of easy to use gestures is very nice.

If you haven't tried gestures on OS X with Apple's touch hardware, I'd highly recommend attempting to use them regularly. I got a fancy Razer Blade 2016 and using the touchpad is significantly worse than an Apple laptop, and OS X has much better native touch support (3rd party software might make Windows better).

I'd say touch interfaces are great compliments to keyboards for text editing workflows (more or less all I do professionally), you don't even need to move your hands off the keyboard to get a mouse (hooray thumbs!).

I would liken getting used to laptop keyboard/touchpad as getting used to a minimalist *nix install. It's difficult at first, but once you get used to it you have a great skillset that lets you do anything you want on almost any computer. Not that you should never nice high-level and user friendly programs, but being able to drop down a level of abstraction and still get the job done is great for understanding how systems work.

[0]: most of my video games suck without a mouse, but that's the only wall my personal use runs into

«I think people don't realize about gestural interfaces is that gestures are hard to do and tiring to do all day.»

To be critical, this is a biological feature much more than it is a bug. Our bodies don't have very good feedback mechanisms for the small accumulative damage from fine motor control, which leads to things like RSI and Carpal Tunnel with very few in-between indicators.

In a way, moving to a more gestural interface that feels physically tiring is a possible means to better overall ergonomics because people actually start to get real feedback (tiredness, soreness) telling them to take a break every now and then when they are physically straining themselves.

>Typing is hard to beat in terms of bandwidth / strain

We type only because we can't talk to computers (although it starts changing). Voice control would easily beat typing.

So long as there's not another human being within earshot of you which is... rare for computing contexts.

I don't see this as a problem. At some point voice recognition will be able to distinguish one individual from another by voice. Also with directed/implanted microphones you might be able to just whisper instead of talking.

Really? Go walk around Facebook's ridiculous one-room office building, experience the ridiculously distracting noise level, and then tell me you "don't see this as a problem" when it's exacerbated by voice controlled computers. That sounds awful.

Really? Voice input is faster for entering text. However, I can't imagine editing text by voice would ever be faster than using a keyboard.

I believe it's just a matter of proper voice interface and commands. Most probably it would require elements of AI for context awareness.

Phonetic chording keyboards are faster than typical speech, they just take a lot of training to learn. A regular keyboard with extensive custom shortcuts for common words is also potentially faster than speech, but not nearly as ergonomically friendly as a chording keyboard.

Keyboards are also much better than speech for making very precise inputs. You’ll be hard-pressed to invent a speech interface which outdoes vim (for instance) for efficiency at general text editing tasks, even if you have a general AI interpreting the speech commands. Even just for straight typing computer code, mathematical formulae, musical notes, poetry with precise constraints on punctuation & spelling & layout, or e.g. controlling a video game or CAD program, a keyboard is much more effective than speech. A keyboard input can also have real-time response to partial input, like autocomplete, etc.

Perhaps most importantly, typing on a keyboard doesn’t distract/leak the content of your typing to everyone in earshot.

With proper AI editing by voice will look different -- probably more like talking to a smart assistant rather than "cut here paste here". Keyboards and mouses will probably remain for some particular tasks (e.g. math formulas or CAD design) although it won't be the mainstream input method -- more like a helper device.

A voice interface interrupts my internal dialogue. I can't think and talk at the same time. So I would disagree, I prefer to let my hands do the talking.

> Our futuristic starship control panel should be a custom-manufactured physical interface instead of a cheap glass panel

I disagree. Modern aviation has been moving away from physical controls and towards generic screens. The last gen of fighter aircraft (The F/A-18, later block F-16s, the F-22) have a collection of generic screens with context sensitive buttons on the side for control.

The F-35 takes this a step further and has a single large screen (which is also apparently a touch screen).

In something as complex as an aircraft (or a starship), there are hundreds (or thousands) of different controls. Look at the cockpit of an older aircraft (e.g. a Boeing 747) as an example. Most of the controls in a cockpit are used in certain situations, you don't need to control the landing gear or the brakes when you're at 30,000 feet.

By using generic displays, the relevant controls for the current situation can be shown. Yes, it is more difficult to operate the controls by feel, but this is less of a worry in aviation, where you're usually several thousand feet from anything else, than it is in a car, for example.

So of course, it's all about the expected situation, you wouldn't want a little touch screen that you swipe for the safety on an assault rifle, for instance, because you need to be able to do it by feel in various situations. But a swipe unlock is perfectly acceptable for a phone.

Reply on news.ycombinator.com