Past and Present Futures of User Interface Design | Hacker News Reader

Past and Present Futures of User Interface Design

datagubbe.se

212 pointssedboyz121 comments

I once worked in a design research lab for a famous company. There was a fairly senior, respected guy there who was determined to kill the keyboard as an input mechanism.

I was there for about a decade and every year he'd have some new take on how he'd take down the keyboard. I eventually heard every argument and strategy against the keyboard you can come up with - the QWERTY layout is over a century old, surely we can do better now. We have touchscreens/voice input/etc., surely we can do better now. Keyboards lead to RSI, surely we can come up with input mechanisms that don't cause RSI. If we design an input mechanism that works really well for children, then they'll grow up not wanting to use keyboards, and that's how we kill the keyboard. Etc etc.

Every time his team would come up with some wacky input demos that were certainly interesting from an academic HCI point of view, and were theoretically so much better than a keyboard on a key dimension or two... but when you actually used them, they sucked way more than a keyboard.

My takeaway from that as an interface designer is that you have to be descriptivist, not prescriptivist, when it comes to interfaces. If people are using something, it's usually not because they're idiots who don't know any better or who haven't seen the Truth, it's because it works for them.

I think the keyboard is here to stay, just as touchscreens are here to stay and yes, even voice input is here to stay. People do lots of different things with computers, it makes sense that we'd have all these different modalities to do these things. Pro video editors want keyboard shortcuts, not voice commands. Illustrators want to draw on touch screens with styluses, not a mouse. People rushing on their way to work with a kid in tow want to quickly dictate a voice message, not type.

The last thing I'll add is that it's also super important, when you're designing interfaces, to actually design prototypes people can try and use to do things. I've encountered way too many "interface designers" in my career who are actually video editors (whether they realize it or not). They'll come up with really slick demo videos that look super cool, but make no sense as an interface because "looking cool in video form" and "being a good interface to use" are just 2 completely different things. This is why all those scifi movies and video commercials should not be used as starting points for interface design.

My experience with people trying to replace a keyboard is that they forget about my use cases and then they're surprised that their solution won't work for me. For example:

1. I'm in a team video conference and while we are discussing what needs to be done, I'm taking notes of my thoughts on what others said.

2. I'm working as a cashier and the scanner sometimes fails to recognize the barcode so I need to manually select the correct product.

Now let's look at common replacements:

A. Voice interface? Can't work. I would NOT want to shout my private notes into a team video call. The entire point of me writing them down is that they are meant for me, not for everyone.

B. Touch screen? Can't work. I can type without looking at the keyboard because I can feel the keys. Blindly typing on a touch screen, on the other hand, provides no feedback to guide me. Also, I have waited for cashiers suffering through image-heavy touch interfaces often enough to know that it's easily 100x slower than a numpad.

C. Pencil? Drawing tablet? Works badly because the computer will need to use AI to interpret what I meant. If I put in some effort to improve my handwriting, this might become workable for call notes. For the cashier, the pen sounds like one more thing that'll get lost or stuck or dirty. (Some clients are clumsy, that's why cashiers sometimes have rubber covers on the numpad.)

I believe everyone who wants to "replace" the usual computer interface should look into military aircrafts first. HOTAS. "hands on throttle-and-stick". That's what you need for people to react fast and reliably do the right thing. As many tactile buttons as you can reach without significantly moving your hands. And a keyboard already gets pretty close to that ideal...

Don't forget MFDs from military/aviation/marine interfaces. Buttons on the edges of the screen, and the interface has little boxes with a word (or abbreviation or icon) for what the button does just above each button on the screen. When the system mode changes, the boxes change their contents to match the new function of the buttons. So you get the flexible functions of a touch screen with the tactile feedback of buttons.

Some test equipment (oscilloscopes, spectrum analyzers, etc.) has the same thing.

TI calculators do the same thing.

>Voice interface? Can't work. I would NOT want to shout my private notes into a team video call. The entire point of me writing them down is that they are meant for me, not for everyone.

Heh, I had a weird nightmare about that. I was typing on my laptop at a cafe, and someone came up to me and said, "Neat, you're going real old-school. I like it!" [because everyone had moved to AI voice transcription]

I was like, "But that's not a complete replacement, right? There are those times when you don't want to bother the people around you, or broadcast what you're writing."

And then there was a big reveal that AI had mastered lip-reading, so even in those cases, people would put their lips up to a camera and mouth out what they wanted to write.

There are many times I really wished to use voice interface but in private. Some notes - both personal and professional - I feel I can voice better than type them out. Sometimes I can't type - it's actually a frequent occurrence when you have small kids. For all those scenarios, I wish for some kind of microphone/device that could read from subvocalizations or lip movement.

In a similar fashion, many times I dreamed about contact lenses with screens built-in, because there are many times I'd like to pull up a screen and read or write something, but I can't, because it would be disturbing to people around me, or because the content is not for their eyes.

>For all those scenarios, I wish for some kind of microphone/device that could read from subvocalizations or lip movement.

There's a similar issue with those automated phone interfaces that "helpfully" require you to speak what you want, because you have to restart every time a pet or child screams. In those cases, it's better to have "press 1 for <whatever>", but it would also be an improvement to have it only read from your lips, so you wouldn't have to worry about background noise.

> "As many tactile buttons as you can reach without significantly moving your hands. And a keyboard already gets pretty close to that ideal..."

The DataHand came even closer: https://en.wikipedia.org/wiki/DataHand

but I'm not sre that's good; moving your hands to reach more keys, without looking, brings even more keys into reach. I can hit the control keys with the palms of my hands - and often do that with the palm of the knuckle under the pinky finger - and feel where they are by the gaps around them, similar with ESC and some of the F-keys, and backspace from its shape, etc. I don't know of a keyboard which is designed to maximise that effect, or how one would be.

https://store.azeron.eu/azeron-keypads does this in a bit different (better?) way

and https://www.tapwithus.com/product/tap-strap-2/ in a different way again.

I'm 100% with you on this but I will admit there was one concept / short run project that actually looked like it was on the right track: The Optimus Maximus keyboard[1].

The keyboard itself was not good for a bunch of reasons, but the idea was gold. Individual, mechanical keys which could change their legends to suit the current context. You wouldn't have to memorize every possible layout before using it, and you could change the layout to suit whatever you're currently doing.

The closest equivalent I've seen would be the pre-mouse-era keyboards which could accept laminated paper legends for specific applications. The next closest, tho in the opposite direction, would be modern configurable keyboards with support for multiple layers layers.

1: https://www.artlebedev.com/optimus/maximus/

There's this great article on the Optimus Maximus, and how it directly lead to the now popular Stream Deck (by Elgato, not the Steam Deck by Valve)

https://www.theverge.com/c/features/24191410/elgato-stream-d...

The fundamental problem with a dynamic layout is that you need to look at it to know what the keys are. The one huge, underrated benefit of a static layout is that it's constant across the environments that you use it in, so it's (always to some degree, rarely perfectly) memorised. Qwerty doubly so, because so many people have it memorised. It avoids the problem with the Maximus that in order to take advantage of the dynamic layout, you really want to be able to see through your fingers. Your fingertips by default block the information you need.

I can see the Maximus being useful for switching between and learning new layouts - so if you want to give colemak a try you can, without needing to swap all your keycaps (even if that's possible on your keyboard), or swapping to blanks and forcing yourself to learn everything by heart. But I think the reason you don't see this idea repeated much is that it's self-defeating.

Do you have to be looking at it? My keyboard has blank keys. I used to use vim, which had a modal input scheme. My music keyboard has modes where different keys play different instruments.

I agree that having the legends are good for affordances when learning. But they oddly hurt training. Specifically, they make it harder to remove the visual from the feedback loop. When training typing a long time ago, you wouldn't even look at the screen til you typed it all.

That's exactly the opposite scenario though: the reason you can get away with having blank keys on a keyboard or a modal input in vim is that the inputs never change, you can commit them to muscle memory and then never consciously think about it. That all gets blown away if the interface is dynamic.

But they do change? Depending on the mode of my editor, they do different things.

There is some stability, but not absolute.

Similarly, if I play a game using me keyboard, which keys control my character will be in contrast to what is displayed on the keyboard.

Now, constant change would be bad. But I'm assuming that isn't what is being proposed?

It is, in the sense that, if there's a way to display a dynamic label on individual keys, some UI/UX designers will make use of that to create interfaces that can't be learned, as they change depending on tasks, environments, and from version to version.

Consider that touchscreen is very much this, and ever since phones got them, you can no longer do anything without looking at the screen. Interfaces are not stable enough to learn (and not reliable enough to operate without looking).

Yeah, on that I will agree it would be a terrible idea. The stream deck should be a good example, though; In that I'm assuming that most people do not have those buttons change much?

Amusingly, I'm now remembering the very old video game system where the different games had cards you slid into the controller to indicate what the different buttons did. Seems like that is the general ideal.

The whole point of the Maximus is that it had little displays on each keycap.

> you really want to be able to see through your fingers

As when graphic artists draw on, and interact with, a tablet, while watching a screen, rather than using a tablet display. Similarly, while I enjoy the feel of a thinkpad keyboard, I do wish it did whole-surface multitouch and stylus and richly dynamic UI as well. So I tried kludging a flop-up overhead keyboard cam, and displaying it raw-ish as a faint screen overlay, depth segregated by offsetting it slightly above the shutter-glasses-3D screen. In limited use (panel didn't like being flickered), I recall it as... sort of tolerable. Punted next steps included seeing if head-motion perspective 3D was sufficient for segregation, and using the optical tracking to synthesize a less annoying overlay. (A ping-pong ball segment lets a stylus glide over keys.)

The flux keyboard seems to be a modern attempt at the same concept. They're taking pre-orders, I don't know if they've shipped any yet or how close they are to shipping.

https://fluxkeyboard.com/

reminds me of:

https://www.jwz.org/blog/2016/09/searching-for-finally-got-m...

and

https://www.jwz.org/blog/2021/03/cursed-keyboard-image/

I may have worked for that company, but I came away with a different take.

People are User Interface bigots!

People get used to something and that's all they want. The amazing thing Apple was able to do was get people to use the mouse, then the scrollwheel, and then the touchscreen. Usually, that doesn't mean that you get rid of an interface that already exists, but when you create a new device you can rethink the interface. I used the scroll wheel for the iPod before it came out and it was not intuitive, but the ads showed how it worked, and once you used it 20-50x it just seemed right... and everything else was wrong! People would tell me how intuitive it was, and I would laugh, because without the ads and other people using it, it was not at all.

Now we're in a weird space, because an entire generation is growing up with swipe interfaces (and a bit of game controller), and that's going to shape their UI attitudes for another generation. I think the keyboard will have a large space, but with LLM prediction, maybe not as much as we've come to expect.

I could go on about Fitts testing and cognitive load and the performance of various interfaces, but frankly people ignore it.

Strangely, apple sucks at mice. A multi-button mouse with a scroll wheel is way better than any apple mouse I've used (especially the round one).

That said, the touchpad on some of their laptops is pretty good when you can't carry a mouse, but nowhere near a good mouse.

(I have owned all their mice, all their trackpads, etc)

Their keyboards have gone downhill too. I like the light feel of current keyboards, but the lack of sculpted keys to center and cushion fingers and arranged keys for the hands has really replaced function with form.

all the people who knew these kinds of truth have probably retired. sigh.

The multiple-button mouse predates the one-button Apple mouse by 2 decades.

The one-button mouse paired to a GUI was an innovative solution: Xerox couldn’t find a way to make a GUI work with one button only, as per their design 1983 article on designing the Alto UI. They tried, did a lot of HMI research but were trap in local maxima in terms of GUI.

Jeff Raskin and others from PARC who moved to Apple (Tesler if I recall well) had seen how three buttons brought confusion even amongst the people who were themselves designing the UI!

So Raskin insisted that with one button, every user would always click the right button every time. He invented drag diagonally to select items, and all the other small interactions we still use. Atkinson then created the ruffling drop down menus, a perfect fit for a one-button mouse.

They designed all the interface elements you know today around and for the one-button mouse. That’s why you can still use a PC or Mac without using the ‘context’ command attached to the secondary button.

I think that's all purist nonsense. It's like tesla removing the turn signal and drive selector stalks from their cars, or people that use super-minimalist keyboards.

Sometimes dedicated buttons for certain functions are GOOD.

People playing FPS games use two buttons to great effect. Maybe one button for aim, another for fire. People with MMOs can use as many buttons as the mouse allows. Creative types in tune with their environment can assign buttons to frequently used functions and flow through their tasks.

Yes, there are ways to double up functions and use less hardware. In macos you can use control + single mouse button for a context menu.

And I understand that poorly designed products use buttons willy-nilly and create a mess. Many remote controls are rows and columns of identical dedicated buttons that are lazy designs.

But why extra-minimal? I think it is a kind of designer-induced technical poverty.

Yeah, the Alto had a mouse. The Chipmunk had a scrollwheel. LG Prada phone had a touchscreen. Few remember them though.

> People get used to something and that's all they want.

It's more than "getting used to." Learning to type (or to edit text fast using a mouse) is a non-trivial investment of time and energy. I don't think wanting to leverage hard-earned skills is bigotry, seems more like pragmatism to me. Unless the "new way" has obvious advantages (and is not handicapped by suboptimal implementation) the switching cost will seem too high.

It's not just that people don't want to waste an investment into learning something; that investment can actually enable you way more than a more easily accessible interaction method, and you stick to it because it's _better_.

Once you've learned to use the keyboard property, it's simply faster for many applications. Having buttons at your fingertips beats pointing a cursor and clicking at a succession of targets. For example, I can jump through contexts in a tyling window manager and manipulate windows and their contents much faster with a keyboard, than wading through a stack of windows and clicking things with a mouse.

It all depends on what you're interacting with, and how often. I mostly have to deal with text, and do not need to manipulate arbitrary 2d or 3d image data.

But suggesting that I am simply too set in my ways to ditch the keyboard in favor of poking things with a pointy thing or talking into a box is just too reductive.

I add to your point and to parent's one.

I do use touchscreens when I've got one on a laptop or on a tablet with keyboard and desktop apps. The reason is precisely what you wrote: pushing buttons in front of us at close reach is faster than reaching for a mouse, aiming and clicking. When the screen is at 50 cm or less from our belly it's not hard to raise a hand and use it.

That also builds on a lifelong investment on pushing buttons, physical and on screen with mice. Using the Tab key to navigate a UI, or shortcuts or hjkl is something that only a few people (comparatively) did so it's can not become mainstream.

The last time I truly learned something new was swyping on my phone keyboard, some 15 years ago. It's extremely niche. I'm the only person doing it among everybody I know. I invested some time but the reward is great, especially when holding the phone with one hand or when there are many vibrations and the screen moves a little. A swiping finger never loses contact. And it's faster for me than two fingers tapping.

On the other side, that's another way to cement the qwerty layout because I swype on it and I would have to adjust to a different layout, so why bother?

Finally, voice. I realized that I used to dictate text to my keyboard on the early 10s, then I stopped. It worked well even back then without all the new technology. I don't remember why I stopped but if I did it means that I didn't lose anything. I wouldn't start again now because I think that there are very few keyboards with only local speech to text.

> Unless the "new way" has obvious advantages

I agree with this. The cases are rare. Still there are cases like the current sad state of motion control in video game consoles, where I have to agree to the opposite. Pretty much everyone who's put the time to play with motion controls outperforms those who don't and can play to satisfaction even without aim assist (which is relentlessly ubiquitous, for those unaware). But the tech started out kinda ass, and the Xbox still don't have a built in gyroscope, so the adoption is artificially stunted. The result? The masses still call it "waggle" with disdain.

The scroll wheel is a step up over both d-pad and touch screen. I also had a Creative Zen which has a scroll lane and it was great too. Why? Because interaction was a factor of motion control and it has great feedback. Same with Apple touchpad. Yep you still have to learn it, but it was something done in a few minutes and fairly visual.

There's a reason a lot of actually important interfaces still have a lot of buttons, knob, lights, and levers and other human shaped controls. Because, they rely on more than visual to actually convey information.

> it's usually not because they're idiots who don't know any better or who haven't seen the Truth, it's because it works for them.

The reason we have touch screen phones today is exactly because Apple dared to challenge that assertion. We should not assume that what is out there now is the end goal. Users don’t have a choice they can only buy and use what’s available to them in stores. The second touch screen phones were available the entire market shifted in a short period, but the mantra at the time was just like you have now “physical keyboard are the only way” who knows what could come from people who think outside the box in the future.

I was recommending laptop to someone and the only criteria he had was a number pad and a big screen because he mostly use Excel. I think input method is fairly context sensitive. Touch is the most versatile one as it acts directly on the output, but I still prefers a joystick for playing game, a midi keyboard for creating music, a stylus for drawing, and voice when I'm driving or simple tasks. Even a simple remote is better than mouse+keyboard for an entertainment center (as long as you're not entering text). We need to bring the human aspect of interface instead of the marketing one (branding and aesthetics).

> Even a simple remote is better than mouse+keyboard for an entertainment center

where you going to find a simple remote anymore?

The only things simple are the giant keys marketing dedicated to partners (like the youtube or netflix buttons)

I want a skip forward button!

Apple TV remote has it’s problems but at least it strives to be simple. Magically it also controls enough amplifier and projector (I don’t know how, hdmi signals?) so I don’t need to touch any other remotes on daily basis.

apple tv itself can be controlled over HDMI-CEC with TV's remote, if the TV supports the mappings

probably hdmi-cec

Before anyone bothers reinventing the keyboard I would rather that it were made practical to easily and reliably type accented and other characters on a UK keyboard in all applications. I use English and Norwegian regularly with the occasional French, German, or Swedish word. I have been unable to find a simple method of configuring Linux Mint to support these other than by switching layouts every time I need an ø or and e acute, etc.

I did once get the compose key to work but the settings didn't survive an upgrade and I have been unable to get them to work again in Firefox.

Use character composition. You then type those characters buy pressing compose key (I've set it to Caps Lock) and then sequence of characters. Much easier than switching keyboard layouts, and you can type other non-usual characters like °, µ, €

ø = Compose -> / -> o

é = Compose -> ' -> e

° = Compose -> o -> o

µ = Compose -> m -> u

€ = Compose -> = -> e

https://en.wikipedia.org/wiki/Compose_key#Common_compose_com...

https://cgit.freedesktop.org/xorg/lib/libX11/plain/nls/en_US...

The Compose approach is extremely handy if you need to type several languages (e.g.: Spanish, German and Pinyin).

A wrote a short article on it a while ago: https://whynothugo.nl/journal/2024/07/12/typing-non-english-...

I also keep a handy alias to quickly find how to write new symbols:

    alias compose='fzf < /usr/share/X11/locale/en_US.UTF-8/Compose'

takes a bit of time but you end up being fast enough without changing keyboard layout, pretty great

The compose key defined in Linux Mint's own keyboard settings doesn't work in Firefox.

MacOS developers have solved this problem pretty neatly:

https://support.apple.com/en-qa/guide/mac-help/mh27474/mac

This is cool!:-)

How come this is not the first “tip” on a fresh Mac?

It's very useful but it's sloooow

This is cool!

But I thought you were going to recommend pressing "fn" to switch layouts (I believe you can use either fn or ctrl+space on macOS).

I use to switch from German (for chat/documentation) and English (for coding), and it's quite instant and second nature to me.

Is there a similar trick for non-letter characters ?

Yes, for some of them, but not all.

I've not been able to find a convenient online image showing the characters you get from holding down alt while typing, it may vary by layout, but for me this lets me type:

Number row: ¡€#¢∞§¶•ªº–≠ with shift: ⁄™‹›ﬁﬂ‡°·‚—±

First row: œ∑´®†¥¨^øπ“‘ with shift: Œ„‰ÂÊÁËÈØ∏”’

Home row: åß∂ƒ©˙∆˚¬…æ« with shift: ÅÍÎÏÌÓÔÒÚÆ»

Bottom row: `Ω≈ç√∫~µ≤≥÷ with shift: ŸÛÙÇ◊ıˆ˜¯˘¿

But of those, I only remember €, # (both printed on the key!), ∞, ƒ, ™, π/∏ (thanks to growing up with MacOS classic — Marathon Infinity for ∞, ƒ for folders, ™/π/∏ no idea why), and –/— (en-dash/m-dash, not sure why I learned them, but was one surprise source of compile-time errors around 2010 because they look exactly like - in a fixed-width font).

If you ever used MPW shell, a lot of those characters were part of the syntax of commands and the regular expression parser so it was common to learn to compose ∫,® ∂ etc. The debugger TMON also used them, so they just become second nature, like !@#.

Neat, did not know that. At the time MPW shell was used, it was a little bit too advanced for me — I was only as far as working my way through C For Dummies (or something like that) with a limited student edition of CodeWarrior* around the time REALbasic came out.

* Possibly bronze edition? Whatever it was, it was 68k only.

`~/.XCompose` is your friend.

I frequently input International Phonetic Alphabet glyphs, some polytonic Greek, some Spanish and some Old English. Nothing is more than three key-presses away after an AltGr.

I'll look into that. The compose key defined in Linux Mint's own keyboard settings doesn't work in Firefox.

Thanks for sharing the keyboard story!

I agree that keyboards can be improved, but I think gradual changes—like making them split and wireless—are a better approach. I use a split keyboard myself and can comfortably do development with just 34–36 keys.

If the interface changes too much in a short time, it can become quite a hassle.

My personal prediction is that nothing will replace the keyboard except direct brain-to-computer interfaces. The keyboard is an incredible tool that people take for granted.

Windows Phone (or was it Windows mobile) had an excellent keyboard with caron/accent keys. So e.g. if I they `C` and then caron key it will replace first `C` with `Č`. I was looking for an Android keyboard with same functionality but didn't find one.

Paraphrasing UX expert Johnny Lee:

  UX = P_Success*Benefit - P_Failure*Cost

...and yet with every new generation of tech, it's never surprising how the hype cycle results in a brazen dismissal of the latter half of this fundamental relationship.

I'm actually working on a voice controlled, tldraw canvas based UI – and I'm a designer. So I feel quite seen by this article.

For my app, I'm trying to visualise and express the 'context' between the user and the AI assistant. The context can be quite complex! We've got quite a challenge to help humans keep up with reasoning and realtime models speed/accuracy.

Having a voice input and output (in the form of an optional text to speech) ups the throughput on understanding and updating the context. The canvas is useful for the user to apply spatial understanding, given that users can screen share with the assistant, you can even transfer understanding that way too.

I'm not reaching for the future, I'm solving a real pain point of a user now.

You can see a demo of it in action here -> https://x.com/ojschwa/status/1901581761827713134

I know that most developers prefer keyboard shortcuts when developing software but I prefer using the mouse mostly because I cannot remember all of the shortcuts in a range of different environments.

Given my preference it would be interesting to explore a more tactile interface.

  - a series of physical knobs to skip back and forward by function, variable reference, etc
  - a separate touch screen with haptic feedback for common functions and jump to predefined functions in my code
  - a macro-pad with real buttons to do the above

Other thoughts

When watching videos physical buttons and knobs would be good. I know professional video and audio engineers already use these technologies but i've never tried them myself.

> I prefer using the mouse mostly because I cannot remember all of the shortcuts in a range of different environments.

This is why one of the greatest changes in power user tooling in recent years is the "find anywhere" hotkey, which is now available almost everywhere.

Mouse interaction is slow and hardly a panacea for finding features buried in menus. "Find anywhere" type interactions with fuzzy search allow you to use they keyboard and highly mnemonic abbreviations to turn up what you're looking for. With a few exceptions, I tend to lean on them even for things that I use regularly, because it's easier to learn which few keystrokes will turn up the option I'm looking for than it is to rebind a fresh hotkey in each environment or, as you say, memorize the built-in one.

Fun anectada. Was launching Remote desktop app which was unsurprisingly called:

"Microsoft Remote Desktop"

by just typing "Remo" in to a spotlight. Then one day it stoped working. I thought i was going crazy, because i dont remember that i uninstalled it. Spotlight was returning no results. Then i found out that someone at microsoft thought that it was a great idea to rename the app to ...

"Windows App".

What the hell were they smoking?

I wish the 'find anywhere' instead expanded and highlighted the correct place of that command in the menu. In that way, it would teach me gradually a mental map of the shape and set of commands I have available, ie I imagine it would let me notice 'neighbour commands'. I realise this is not the most efficient way to do this, but it would be 'incremental on the job learning'.

macOS does this when searching for a command in the Help menu - it will open up the context menu and highlight the place where the command is so you know where it is next time.

> "find anywhere" hotkey, which is now available almost everywhere

Almost everywhere? I'd love to see a list of other examples.

The only place I can name is the vscode Ctrl-Shift-P thing, and in that case it's a wholesale replacement for an explorable/discoverable UI (i.e. traditional menu bar).

Sure there are search boxes in other places, but usually that's literally for finding things, not for performing application domain commands/actions/manipulations, which is what I understood the parent to be describing.

> one of the greatest changes in power user tooling in recent years

Alt+x in Emacs was here for ages. Even the command prompt in vim follows the same pattern. And while it's useful, I still prefers to bind commonly use commands to keybinding, especially if there can be prefix.

There's a reason I didn't say "innovation"—I knew that people would immediately point out it's been around forever. What's new is that it's in mainstream tooling.

If OSes were optimized like RTS games, maybe the mouse could be plenty fast. Something like https://charmstone.app/ but for many actions.

And what is the "find anywhere" hotkey that works everywhere?

OP is talking about the "Cmd + P" aka "Run command" command. Ideally it should list the shortcuts of the commands you run so next time you can do that directly.

The MacOS-wide equivalent is Cmd+Shift+/, which searches all menu items. KDE has a similar idea built into the global menu.

I use the mouse a lot, even for (typing) coding.

I'm pretty "fluent" with navigation shortcuts, things like Ctrl/Alt/Shift combined with arrow keys, PgUp/PgDn/Home/End etc. and I do use them extensively. And yeah, constantly switching between the keyboard and mouse with my right hand is a bit annoying.

But still, in many cases, using the mouse is just faster. For example, jumping to a specific position in a source code file, scrolling and clicking gets me there much quicker than navigating with the keyboard alone.

(This is also one thing I really hate about using terminals: you can't just click to move the cursor quickly! Editing part of a long string without spaces is a pain in the ass, and it's something I have to do surprisingly often.)

When it comes to shortcuts, I prefer one- or two-key combinations whenever possible. Three-key shortcuts, however, depend on their layout: many just aren’t that convenient. Sometimes I’ll just click through the menu manually, even if I know the shortcut.

> For example, jumping to a specific position in a source code file, scrolling and clicking gets me there much quicker than navigating with the keyboard alone

I say this with the intention of providing context, not to say the way you do things is bad. It's all user preference in the end and there is no wrong way.

Lots of folks consider your "fast" example with a mouse as their "slow" example that forced them into learning more advanced features of their editor. For example. Most Vim users can get to any character or partial string, or parameter, or line, or paragraph, or function start or what you have within three quick keys on their home row. They do this quickly, and can immediately start doing other things right after because their hands never moved.

The mouse is fast because people don't need to memorize things. The keyboard is fast because the keyboard is fast.

It's like the old joke from the movie Heist. "What do you mean you don't like money? That's why they call it money".

No offence taken; there's always room to learn. Just curious, since I never use Vim, how exactly one navigates to (and/or selects) "aaa" part in the following string with just three keys in Vim?

    url = "https://example.com/keyword=aaa&name=john"

So many ways!

fa;

(f)find (a) (;)repeat

That would be the generic vim way. I could mash on semicolon to get to each instance of "a" in the line. Here's another.

2fa

2 (2nd instance) f (find) (a)

Most people use a plugin called easy motion instead.

You type some two character key command to start it. For me it's "ff".

After ff, i type any two characters. It will then highlight each place in the document that start with those two characters (think an inline table of contents) that I can then select.

https://github.com/easymotion/vim-easymotion

I write all this knowing it looks and sounds like madness, so again, don't take this as anything other than someone explaining their madness, but once you learn all this stuff (it takes about a month) you realize your mouse is actually the slowest way to do it.

if you are above, move there with three keys: t&b

select: t&vb

if you see below you'd do F&b, F&vb

I'd do `/aaa<enter>cw<replacement text>`

> I really hate about using terminals: you can't just click to move the cursor quickly!

alt-click?

Doesn't seem to work for any emulators I have (cmd, Windows Terminal, Git Bash, etc.) :(

Use a different terminal and/or editor. Mouse support is absolutely a common feature and has been for a while. xterm, konsole, tmux, emacs, GNU screen, etc., etc. all let you do a variety of things with the mouse.

For instance I use helix and can move the cursor with the mouse just fine in WezTerm, iTerm, and Apple's Terminal.app. tmux has a mouse mode where you can select and scroll with the mouse.

There are compromises that should have never been made. Just to please the eye. The Blackberry keybord worked fantastically well. It was killed because Steve beleieved that touch screen is always better. Then there are the knobs in cars replaced by touchscreens. I hope there are no attempts at replacing physical keyboards by touchscreens in laptops. I cannot type on touchscreens. I love keyboards. Especially my mechanical keyboard. Love it.

I think Steve (et al, I will not attribute everything to one person) did have a point; it's 2025 and an on-screen keyboard makes different inputs easier, for a lot more languages than physical keyboards can, without needing separate production lines.

There's multiple input methods; typing, swiping/path writing, braille, dictation. It's not for everyone, sure, but for others it's ideal and preferable over a physical keyboard.

> an on-screen keyboard makes different inputs easier

Sure, it can certainly make different inputs easier. But you'll be hard-pressed to convince me that those different inputs are better specifically because a touchscreen doesn't have the kind of physical feedback that comes from feeling the boundaries of different selections (eg, letters/characters) for that input.

With a keyboard, I can center my fingers without looking at them. I know exactly what input selections are nearby and in what directions; and I know exactly when my finger has crossed the boundary of one and is in a null-zone or is on another input. For many keyboards, I can even feel the embossing of the individual input selection (character).

> It was killed because Steve beleieved that touch screen is always better.

No, it was killed because people much preferred Steve's product than Blackberry's. In large part thanks to trading off a 5% of keyboard effectiveness in favor of 50% additional screen space.

> I hope there are no attempts at replacing physical keyboards by touchscreens in laptops.

There have been many. Laptops that are just two touchscreens hinged together. Every review absolutely demolishes them for the obvious.

> For almost half a century now, we haven't really managed to come up with something better, and that's not for lack of trying.

In contrast, my impression is that deprioritization of trying is a defining characteristic. Patents - "yeah it's neat... you can't sell/buy/have/buildon anything like it". Narrowing optimization - "thinkpad is 2-key rollover, because business apps don't use 3, and it saves some cents". Software design-space badlands - "to tweak that, reimplement the full stack and apps". Unicorn dreams - "mass-market or nothing... most often nothing". So instead of a rich ferment of DIY and multi-scale commercial exploring for diverse viable niches, we have an innovation monoculture desert of "wait years for bigco to maybe find incredibly-challenging mass-market fit and afterward backfill niches". So a "we can't sell it because patents so here's a kit"-crippled but still creative AR/VR DIY community hits "Facebook bought Oculus for how much!?!" unicorn dreams and dies. So... etc.

More than anything, the interface which I want to normalize is holding a laptop as a book and using the touchscreen (and optional stylus) on one side, and the keyboard (say for drawing shortcuts/modifiers) on the other side.

Still surprised that this wasn't a standard for say the Voyager e-book reader.

Hopeful that the Lenovo Yogabook 9i will help to popularize this (and if it had Wacom EMR, I'd have one and be working on such concepts) --- annoyingly, my Samsung Galaxy Book 3 Pro 360 is just a bit too large for this, and the screen is so impossibly thin, trying it makes me worry about breaking it.

The next biggest shift in interface is moving from a tactile input -- keyboard, mouse, touch screen, ect. -- and visual screen output to none tactile input -- voice, brain implants, ect. -- and none visual output mostly automation or multistep tasks. Some attempts so far haven't been successful, i.e. Alexa and Siri, others look promising like OpenAI Operator, and it exists in sci-fi like Iron Man's JARVIS, nonetheless, it is definitely the future.

I worked on a browser automation virtual assistant for close to a year -- injecting JavaScript into third party webpages like a zombie-ant fungus to control the pages. The idea of tactile input and visual output is so hard coded into the concept of an internet browser, to rethink the input and output of the interface between the human and the machine, everything becomes hack.

After a decade working in UI, it was strange to be on a project where the output wasn't something visual or occasionally audio, but rather the output of the UI was automation.

I’m highly skeptical about voice for most interactions. It’s inherently inappropriate in most public settings.

+1. I see it being helpful for the differently abled but for everyone to just speak out loud their every action would drive everyone nuts in a public setting. Not to mention I can type “code” much faster than if I were to speak it.

There are a few ideas about subvocal recognition kicking about which might change that. If your voice assistant is in an earpiece that can (somehow) read what your vocal muscles are doing without you actually needing to make a sound, it makes it practical to the extent that it could become the default. There's a lot of ocean between here and there, though. Particularly in the actual sensor tech. That's got to get to the point where you can wear it in public on a highly visible part of the body without feeling like a loon, and that's not trivial.

That maybe makes it vaguely less anti-social but still imprecise and frankly invasive. Typing by comparison is great. You can visualize the thoughts as you compose something and make edits in a buffer before submitting. The input serves as a proxy for your working memory. Screenless voice interfaces are strictly worse.

That assumes you have to be right first time, that you only have one chance to submit your buffer. We don't make that assumption when we talk to other humans.

We were putting this into classrooms where teachers were speaking all day anyhow. The system completely automated teaching tools, smart boards, and browsers. I don't think it gained a lot of traction, nonetheless, the company raised $100,000,000 to focus on the automation part of the product as a vertical.

My point is that as a UI developer, I was moving from all output to screens to output which is automated tasks. There are different types of output and they almost all relate to the senses which is the were the interface between the human and the machine exists. For example, to screen, eyes, to sound, ears, and haptic feedback on mobile devices, touch. Because in my space, the browser, I was using JavaScript and browser APIs the same but the end result was different.

Automation as an output is fundamentally different from all the UI I built the decade prior.

What’s the value prop here? That seems antithetical to actual education.

Shit writing on a white board has better physical feedback than some janky smart board.

True, but we could move to lip-reading for that:

https://news.ycombinator.com/item?id=43400636

On Star Trek TNG they have what appears to be a bank of Fujitsu Eagle disk drives on the bridge to the right of the elevator. Looks like 10 for a total of 4.7 GB worth about $100,000! Those are so noisy, you'd think the'd put them in another room so they didn't have to work next to them.

https://en.wikipedia.org/wiki/Fujitsu_Eagle

https://www.syfy.com/syfy-wire/chosen-one-of-the-day-those-c...

Hah, I never made the connection :).

Though on the show, the computer core is far from the bridge, and it only makes cute sci-fi noises. As for the thing you refer to, I don't recall it being ever revealed what those are. They kind of look more like equipment lockers, but if they were, the placement doesn't make sense.

“In a commercial for the Alto, we meet a man - some kind of upper middle management, presumably - going about his daily business. He works in a spacious private office and, using the Alto, he can read and send email and produce laser printouts. Eventually, the Alto conjures up a high resolution image of flowers. The man wonders why, and the computer replies - with text on screen - that it's the man's wedding anniversary. "I forgot," says the man, to which the Alto replies, "It's okay, we're only human."”

Sounds like fruit intelligence features

> Imagine having to raise your arm to swipe, pinch and tap across an ultra-wide screen several times per minute. Touch works best on small surfaces, even if it looks impressive on a bigger screen.

I regularly find myself wishing pinch zoom were available on my large multi-monitor setup, even if i only used it occasionally, i.e. to augment interactions, not as a replacement for other input methods. As a (poor) substitute, I keep an Apple trackpad handy and switch from a mouse to trackpad to do zooming. Sadly I’ve found not all macOS apps respond to Magic Mouse zooming maneuvers.

Surely i'm not the only one but I've always felt when it comes to the discussions about replacing the computer desktop experience there's some serious historical blinders on.

Work surfaces like the desk as an idea has existed and had been refined for millennias as human interactive spaces before being borrowed and codified to work on computers.

it's not bad to look into but i don't think its surprising to find that human physiology hasn't changed, and so the interface environment ideas are still useful for similar situations

I don't agree with the fact that touchscreens are necessarily cheaper everywhere: take cars as an example. If a touchscreen makes me more prone to car crashes, because I have to look at it more often than with knobs, in the end it costs more. May take a while to fully reach better interfaces. Sitting at my desk, otherwise, means I can look at a touchscreen.

It's cheaper for the manufacturer. It shifts the UI work from expensive hardware design and manufacturing, that needs to be in-sync with the rest of design and manufacturing of the car, into purely software realm that can be outsourced to some cheap noname company. All the hardware complexity goes away, you only have a glass slab to fit somewhere.

Though arguably, the starker difference is with home appliances, where touch interfaces replaced buttons. There, there's no dynamic display (at best, only a light behind a button turning on or off) - so you don't have touch screens, you have a much simpler touch recognition technology, that has zero moving parts and can be pretty much etched onto the board, which is a significant saving over mechanical buttons. And it looks cool, which helped with marketing initially.

that's correct, however, as consumers, my opinion is that we (that is, we that share my point of view, obviously!) should try to buy what we prefer regardless of what's cheaper for the manufacturer. If possible of course, we should spend more for something that in the end we think it's better and not fall for flashy designs that makes it look like you're driving some kind of futuristic car.

About home appliances, specifically kitchens, in my country (Italy) there has been a widespread wave of electric kitchens with touch interface, which are awful if you do intense cooking: it all becomes covered in water and other substances making it harder to cook. Maybe cooking quality and ease is not that relevant, considered the other benefits of having a touch screen, i don't know: but the design itself, in my opinion, expressed as beauty that better vehicles functionality, is "broken", that is, breaks this definition.

Of course this is all my biased opinion.

Different UI paradigms? Then Dynamicland [0] communal computing by Bret Victor [1] comes to mind.

Anyone knows other examples where truly different UI paradigms are involved?

[0] https://dynamicland.org/

[1] https://worrydream.com/

This is a lot of fairly interesting build up and background leading to a very short and shallow takedown of voice control that's all about audio as IO for a couple practical reasons.

No discussion of whether natural language will be a powerful addition to or replacement of current UI, which of course can just be text you type and read.

AI chat will replace the huge number of menus and complex interactions in many content development applications. In VSCode, for example I rarely touch 98% of the UI anymore. I just use Copilot. I no longer need large, complex chunks of the IDE, like the old refactoring code, or the formatting tools. Similarly, in 2D and 3D content creation, isn't it clear that we'll move to some iterative process using AI chat instead of millions of mouse clicks? There are clearly still issues with this, but there were issues with the early GUI tools too. I think huge swaths of complex GUI interfaces will disappear over the next few years. I know many folks who have spent decades in the VFX industry learning how to use Maya, video editors, paint tools, and proprietary tools, and lots of developers who built custom pipelines who might think it will take a bit longer. But it is coming for sure. Maybe you can optionally use voice, but that's not the important bit. This feels like it is going to be a significant shift in how we use computers, and it is already happening.

Why use a 3 key refactor shortcut when you can describe what you want in only a few back and forth conversational chats?

Because what you describe in the chat has a much more significant impact. Sure, you could mimic each little mouse click and short cut, or you could say "make the arms longer", or "write a test for the new endpoint".

And why do you think delegating the responsibility of tasks to the AI is going to end well?

"Make the arms longer" could lead to way too many polygons, or messed up UVs, or will create arms that don't have edge loops, or have them but along the wrong axis, or etc...

"Write a test for the new endpoint" is going to lead to "ok, now make the test pass" which is just going to make the test return true, or return a false positive by checking that a function exists instead of calling the function, or calling the wrong value, or changing the code to fit a test that doesn't do the thing it's supposed to but at least the test passes now, or etc...

The only thing "AI" is doing is either spitting out something that already exists from google, or making a mess that will need to be cleaned up later.

I've been writing code since before GUIs existed. Last year, many, probably most experienced programmers were skeptical about using LLMs to help write code. This year, there are vanishingly few who don't use an LLM every day. Words are easiest for LLMs so programming changed first. Images, audio and 3D are next. It really isn't a question whether the change is coming, it is just a matter of time. And, change is coming faster and faster. It isn't perfect, but it is getting better at an alarming rate. It is just a new interface, not a replacement. The new interface allows you to iterate in different ways to produce a result. It isn't about providing the right magic incantation and having a perfect result. Like with current tools, you iterate, taking bigger steps each time and as you learn how to better use the system.

This comment was honestly shocking.

I spent more than 15 minutes trying to think of the first thing to bring up, but there were too many things to go through. I went on a tangent about belief bubbles then thought replying was too taxing, then I realised that not replying may be contributing to the problem of those bubbles existing in the first place. So I'll try to keep my reply brief and succinct enough to not lose my mind by just responding to only some points you brought up.

> Last year, many, probably most experienced programmers were skeptical about using LLMs to help write code.

Firstly most experienced programmers were sceptical about using LLMs to help write code, and still are.

I am qualifying a programmer as "experienced" not as those who have held a job title for 15 years, and not as those who have bounced around between projects until they release ( tangent: a project that comes out with 10 years of support is not successful just because it launches. It has to have what it takes to actually be able to be supported for the whole 10 years and not require a rewrite within 2 years), but those who have worked on enough varied projects while providing support for 10 years to understand which trade-offs are worse in the long run, and why.

Most "experienced" programmers did try LLMs, saw the hallucinations, code churn, and logical errors and decided to stay away from LLMs.

But individual anecdotes aren't that useful which is why we need to look at qualitative results over quantitive.

Large groups started doing their own studies which have just begun to start outputting qualitative results in the last year.

The results show that using LLMs in projects leads to more bugs, increased code churn, and less understanding of a codebase's internals, each of which have been shown to reduce the health of a project (for both codebase and developers).

And because of those results, experienced programmers who were on a wait-and-see path further away from using LLMs to help write code.

> This year, there are vanishingly few who don't use an LLM every day.

It's worth pointing out that the following two statements are both true:

1. the number of experienced programmers has risen 2. experienced programmers are vanishing few

1. Is true because there are more programmers than there were 10 years ago, and the number of programmers could be qualified as experienced has also risen

2. Is true because the experience and quality of programmers looks less like a normal distribution and more like a log-normal distribution. (The quantity goes up but the percentage remains low, so be weary of quantitative claims).

The "vanishingly" few are the experienced programmers mentioned in previous point.

> Words are easiest for LLMs so programming changed first.

Programming isn't just using words, it's using logic. Cobol showed us that spoken languages make poor programming languages.

What LLMs were good at was reciting common boilerplate, since it's always the same words in the same order. It didn't need to understand the logic behind those words.

But as soon as it steps out of that wheelhouse, it falls down. (more on that in the later tangent)

> Images, audio and 3D are next.

Much like with text generation: images, audio and 3d fail at the same points.

Image generation can recite textures, but has no understanding of composition.

Audio can recite phonemes, but has no understanding of delivery.

3d can recite a shapes or details, but has no understanding of a modeller might use want to use a certain techniques in specific places so the shapes end up being problematic to work with.

> It really isn't a question whether the change is coming, it is just a matter of time. And, change is coming faster and faster. It isn't perfect, but it is getting better at an alarming rate.

Actually no, we're spending more resources trying to give each model more parameters to use, more time to train and a larger corpus to ingest and we're getting diminishing returns. Neural scaling laws are not exponential, and they are not linear, they are logarithmic.

-------------------------

Tangent:

Investors were hoping for exponential returns, but when it became obvious that wasn't going to happen, the large AI players suggested that linear increases in compute and training would deliver linear results. (the implication being put all your money in our pot because it will be the largest and so return the best results)

Every time they announce massive productivity gains in programming and yet every time questions come in once the reports have been scrutinised and they push out a new report to attempt to drown out the questions, all to keep their own valuations high.

Recent examples with OpenAI:

- OpenAI released a bunch of results stating that their models could solve about 49% of test cases in an external software engineering benchmark (SWE-Bench based on issues in 12 open source python codebases). But after an external team manually verified each test it turned out that it could only solve about 4% of test cases and would have created more work for programmers had the other suggested changes been applied. See: https://www.youtube.com/watch?v=QnOc_kKKuac

- Then, OpenAI's researchers built a new benchmark (SWE-Lancer based on SWE tasks on UpWork), tested a bunch of models and checked the results. Turns out if you read the paper that only half the questions were asking the LLM to make a change to the code. The other half were management questions picking which solution to go with. (of course that's going to be easy to game since one dev can say: "Hey this isn't very good but it'll work for now" and another dev can say: "This will solve all of our problems" and the LLM will pick the second solution every time). It's not checking the logic in the code, or picking the best solution for what the project needs moving forwards, it's just picking the most confident sounding message put forwards by others. See: https://arxiv.org/pdf/2502.12115

and so on...

The other big players are just as bad. (I'm not even going to get into how Anthropic's CEO is inflating the bubble by claiming that AI will be writing 90% of code in 3-6 months to gullible investors.)

-------------------------

> It is just a new interface, not a replacement. The new interface allows you to iterate in different ways to produce a result. It isn't about providing the right magic incantation and having a perfect result. Like with current tools, you iterate, taking bigger steps each time and as you learn how to better use the system.

Yes it won't give you the perfect result, but that doesn't matter if you can iterate towards it.

The problem is that it's actually worse for iterating towards a quality result.

An LLM will suggest something for fixing one issue, then when it gets into another issue that touches the same file, it leads to having to remove a whole chunk and rewrite something that could have just been written the first way.

The results from those qualitative studies are showing that diffs are coming in more frequently and they're coming in larger. This is increased code churn (which as explained before is bad for the project's health).

Though we seem to disagree, perhaps less than you might think, I'm very happy that my comments made you think for so long and write so much in response. It will be interesting to see how this all plays out over time.

Speak for yourself; the other tools aren't going away, just because you're less effective at using them

I just wish I had a hotkey that would turn everything on my screen to Liberation Sans Font, 14-pixel, Boldface and keep it there until I possibly hit another hotkey to reverse it. If the occasional item needs to be emphasized, then it can be underlined.

> I instead urge UI designers to consider the powerful concepts of consistency and familiarity just a bit more often than they presently seem to do.

I've been developing UIs in some form or another for almost 20 years and this really hit home for me. I often use the analogy that UI/UX designers are like interior designers and architects rolled into one while developers are the general contractors and tradespeople (you need to pretend "software architecture" isn't a thing for the analogy to work). In my experience, I could ask a general contractor to come up with a design/layout for a kitchen and I would probably be satisfied with the result. That contractor has built hundreds or thousands of kitchens. A good GC will know how to lay out the space efficiently given the dimensions of the room. You won't end up with weird issues like needing to open the oven door in order to access a cabinet drawer (something I'm dealing with in my current apartment). It probably won't be flashy or unique or have a "wow" factor. In fact, it would probably be so unremarkable as to be forgettable, but it would feel immediately familiar to anyone who comes to visit and function perfectly as a kitchen.

I try to push for this type of approach in my current job. I don't want to try implementing novel design ideas. Novel design ideas don't have reference implementations or accessibility guidelines. I think familiarity is the best UX. Are the familiar approaches always the best ones? Maybe not, but I don't want to be the trailblazer. Whenever I encounter a novel design, I ask the person who came up with it to find me 3 other apps that implemented something similar. If they can't, I suggest a more idiomatic solution. Sometimes people push back, but once I start asking very specific questions about how this thing is supposed to behave, they usually end up deciding to drop it and go the idiomatic route.

I can't say for sure, but I'm guessing a lot of the articles and blog posts out there complaining about the state of user interfaces aren't written by the folks that have to implement them. There are so many variables involved in accommodating for user interactions with basic controls, let alone complex ones (if you've ever built a custom <select> component that meets all a11y requirements, you know what I mean).

So to end my rant: just use the dang controls we have available, don't try to reinvent the wheel, and focus on making your product really good at what people are paying for it to do. I wouldn't spend extra money on an application just because they came up with a novel alternative to a slider control, and I'm guessing most other people wouldn't either.

I just wish I had a hotkey that would turn everything on my screen to Liberation Sans Font, 14-pixel, Boldface and keep it there until I possible hit another hotkey to reverse it.

At least for games, design everything to be useable for mouse input only, if you want my money. Mind you, do not play FP shooters and the like.

Mechanical/Analog will always win in the end

I know one thing for certain: The future ain't what it used to be.

I, for one, am glad we didn't end up with monkey hands/minority report screens that we interact with by flailing our hands and arms in front of.

Pretty superficial.

And no mention of the much-hyped "Minority Report" UI that failed spectacularly for obvious reasons.

Reply on news.ycombinator.com