Building the First GUIs | Hacker News Reader

Building the First GUIs

computer.rip

96 pointszdw95 comments

My sense is that GUI evolution has stagnated because the driving force behind Engelbart's (and others) thinking, the idea of a computer as an augmentation device that empowers individuals has been essentially abandoned.

Peak "augmentation" seems to have been (believe it or not) the Microsoft era, complete with its "productivity suite" (except productivity has gone missing) and killer apps like the spreadsheet that is still a computing interface out of this world.

The demise of the MS monopoly came about by the exploitation of opportunities offered by interconnected computers with zero regard to lofty empowerment ideals. The role of the computing device has become to offer just enough attraction so as to extract value from the individual using it. Achieving that at scale has required the dumbification of the UI, removing any cognitive stress. Except, no pain - no gain.

If the hypothesis is correct a next generation of important (G)UI's may come about if and when the much delayed harnessing of the networked universe to augment individuals starts taking place. It might be driven by the challenges of the new stage we are in. Think of concepts to address information overload or managing online personas. Or it might facilitate emerging possibilities, eg interacting with local algorithmic agents etc.

> and killer apps like the spreadsheet that is still a computing interface out of this world.

I wish the expressiveness of tabular data representations was more obvious to more developers. All you have to do is sprinkle in a little bit of relational calculus and you can model literally anything as an xlsx document. Then, unlike virtually every other programming environment on earth, you can hand that document to any other non-technical human in the same business and they will immediately understand what you are trying to show.

9/10 times the spreadsheet will give the business what it wants, assuming you invest enough time in developing a decent one. The only reason you should add code on top of that is if you identify additional value-add with persistence, validations, systems integration, concurrent access, fancy UIs, etc.

> I wish the expressiveness of tabular data representations was more obvious to more developers.

On the other hand, I wish spreadsheets exposed a more developer-friendly feature set. It wouldn't take much: a sane language for formulas, some form of procedure/subroutine functionality with document or sheet scope, separation of semantic content and graphics.

Part of what's infuriating about Excel is how it's a local optimum not too far away from an even better local optimum. I could see myself actually using spreadsheets more, even professionally, but those poor things are so badly misused it's nauseating.

A mostly forgotten these days aspect of Office is that you can access pretty much everything in it using COM - including both extending and embedding Excel, where you get full control over the data inside the spreadsheet.

And even with just VBA, I've seen some impressive spreadsheet applications (in fact, some even were versioned and had official SDLC process in some companies).

For me the ultimate enhancement to excel would be to directly integrate some SQL dialect into the application.

Assume you had a few worksheets and you want to produce a new one based on some projection of those. You could press some hotkey "New Worksheet from Query..." and then type in the SELECT you want to use for the projection. The column headings could be optionally specified with the appropriate SQL syntax. The schema for the internal SQL dialect could be dynamically generated by inferring table name from sheet name, and column name from the first row in each. You could even have this re-evaluate in real-time, so any changes to base sheets would instantly update the projected sheets.

Soooo VBA? https://docs.microsoft.com/en-us/office/vba/api/overview/exc...

What I'm suggesting is the opposite of VBA. There should be no difference between the language you use to script the worksheet and the language you use to write formulas in the worksheet itself.

VBA, COM and things like that rest upon the assumption that the worksheet is just a complicated data structure like the DOM.

Want to take Excel seriously? Let's commit to that. Take sheets as the computational model and build from there. A natural operational semantics for them is a graph rewriting system, same as ML and Miranda.

What you describe as the "productivity" vs. "dumbification" seems to me more like the distinction between a professional environment and an entertainment medium, respectively. There is no point in comparing Windows 95 tools to IPhone apps without taking into consideration the upside down target audiences.

I'd argue that GUI evolution has all but stagnated: The shift from specialized tools for individuals to mass adoption brought us much more new ways of interacting with our devices. The adoption of personal computing devices in the last several years has been unprecedented. We'd still be reading paper newspapers, watching TV and using the landline network for communication if it were not for the recent GUI evolution.

The main problem I see is the trend to remove power user features altogether instead of having them be hidden in the settings. Ease of use and customizability can get along just well, but are rarely to be found. So your statement about augmentation is right anyway.

Edit: I think that different OSes, different devices for different target audiences is not an actual problem. The uses of a computer can be so vast. The trend to integrate smart devices has only begun and requires different usability approaches. We should not compare Excel UX to Tinder UX

On second thought it is true that there has been a fairly major UI breakthrough since Engelbart: The touchscreen. It is arguably still underutilized, mostly an ersatz mouse function. What would be the paradigm-shifting multitouch application? My guitar, with zero AI and a UI going back some millenia is far more sophisticated in its adaptation to ten digits (Yes, I know Django could do with two :-)

You've sort of encapsulated a large part of my underlying philosophy there... that computing evolution has stagnated because of a loss of motivation to take on the hard problem of making complexity accessible. I like to think that everything I write sort of converges on that idea, but I don't tend to say it outright because it's difficult to fully explain and last time I edged too close to it I got more angry emails than when I spoke ill of RMS.

There is still vast amounts of money to be made using the scale-power of web/data tech developed in 2000s to colonize various legacy markets still stuck on even older tech, like banking (Stripe, Nubank). Next-gen UI will come when someone proves out a business model to drive the investment needed to develop it. (I think we are in agreement here) Also the current macro climate is not really conducive to R&D, you'd have to be an idiot like me to invest in advanced PL research startups when GOOG did 21% CAGR 10Y historical

I don't think of augmentation specifically in terms of productivity, but more in terms of an external brain.

I'd say peak augmentation came with having Wikipedia available in my pocket or from a smart speaker. There's still so much more that could be done. If anything, word processors and spreadsheets are a local maximum and I think we are probably stuck on another one now.

> In the eyes of most, the next major step from the Macintosh was Microsoft Windows.

No, that would be Atari STs GEM OS, or the Amiga’s AmigaOS. Or the Acorn Electron. Windows was laughing joke amongst non-PC users for the entirety of the 80s and the first half of the 90s. Even Tandy produced a better frontend to DOS than the early versions of Windows.

At the end of that paragraph:

> It was more that early releases of Windows failed because they were inferior to other DOS GUIs.

I think the author agrees with you, they’re just noting that this is how many people today (including myself, prior to having read the article) perceive the evolution of GUIs.

Acorn, Atari and Amiga were not DOS machines. There was a lot of variety of operating systems and computer hardware back in the 80s. It wasn’t until the mid 90s when the industry “standardised” (in a de facto sense of the term) on IBM-compatible machines. Before then most didn’t even run Intel CPUs let alone DOS.

I made a practical choice to focus on the IBM PC compatible GUIs because I am most familiar with them and I think they are some of the most poorly known due to an assumption that Windows was always the dominant option. And yes, I think it's fair to say that Windows 1/2 failed due to its lackluster comparison to other DOS options---DOS had significant market share that should have made Windows a top competitor, but it was not, as DOS users were choosing other options.

That said, you are absolutely correct that there were multiple competing microcomputer platforms at the time and most of them had GUIs on offer.

It didn't help that graphics on the PC were absolutely terrible. IBM's monochrome monitor was beautiful, but text only. Their color graphics card had one of the worst interfaces you could imagine short of setting pixels one-by-one over, say, a serial port.

The PC world would have been a far more interesting scene if IBM had consulted with or purchased technology from someone like Atari or Commodore.

Don't get me started on audio.

IBM really blew it. Arguably, we're lucky they did, given the market opportunities that they created as a result.

Did you mean the Acorn Archimedes? (The Acorn Electron was a cut-down version of the 8-bit BBC Model B. The Archimedes was the first ARM computer (32-bit RISC CPU, full colour WIMP interface, anti-aliased fonts, etc.).

Ahh yes. Good spot. Thank you for the correction

Is there a growing sense that our GUI/UI paradigms took a wrong step somewhere in CS history and we've pushed the current UI path as far as it will go?

Is it only because I myself am building an alternate OS interface or has this become a popular subject of conversation?

I've had the feeling that back in the 80s through the 90s there was a lot of formal design study of UI and UX. Actual scientific studies were taken of what makes for the most efficient and economical graphical UI.

And then the designers came and threw away so much of that knowledge because they wanted to make something that looks good in screenshots. Visible controls are ugly they said, so all buttons and controls get hidden away under some nondescript icon that doesn't even look like an interactive element (maybe 3 lines in the corner or something). Scrollbars are uncool so now they disappear immediately and you just have to know where to look for them. White space is visually pleasing so just fill the screen with it. Old rules about making buttons look like buttons are completely ignored so the screenshot is pretty. Sure you have to hunt around like a blind person in an alley trying to make things actually work, but it sure looks nice doesn't it?

I kind of disagree, I think this was more based on ad-tech taking over and pages optimizing for SEO and session-length and things like that. The metrics changed when the customer changed as we moved to free software.

It seems kind of a cheat to increase session length by making your page so hard to use that people will spend extra time just trying to figure out the controls.

I don't think that's what GP is referring to.

Ad-tech and similarly myopic business pressures are frequently at odds with usability, and those business pressures ultimately determine what solution makes it to (or remains in) production. Less usable websites and tools are frequently "better performing" when measured by a metric leadership is beholden to, and usability is easily sacrificed if faith in its value doesn't extend to the very top of a company.

I know this is HN and it's fun to pick on incompetent designers, but there are a lot of parallels with the lack of care for security that frequently tops the news on here. Poor usability often reflects company values, not designer capability.

Yes, I meant that priorities and metrics changed, not that they intentionally sabotaged one metric in order to "cheat," and directly boost another metric, it just wasn't a priority anymore to optimize for usability. In fact, when you look at it from a market/systems level instead of any individual product or company, it doesn't even need to be one designer or product or company moving A-to-B from "usable" to "moar white space." It could be a totally different company who never optimized for usability who comes in and steals lunch.

>there are a lot of parallels with the lack of care for security

Absolutely, 100%. Same idea.

They were probably formally studying people who had never seen or used a GUI before. Visible controls were a great crutch that helped get us to the point where we don't need them anymore. Because at the end of the day, content is king. For example, Netscape Navigator 4.0 had a big button labeled Find with a little binoculars icon. Today, everyone knows about CTRL+F. So we can use that screen space for the thing we care about, which is content.

We still have much further to go. My main display doesn't have a single UI element on it. It's a terminal in full screen mode. Except devoting 100% of the display to content wasn't enough for me. I wanted a third dimension of content. So I modified the terminal software so that I could control transparency by shift + scroll wheeling which means I can play movies in VLC running in full screen mode underneath the full screen terminal. That's my design. Terminal and television. The most important use cases of the most important devices of the twentieth century.

> Today, everyone knows about CTRL+F.

Not everyone knows that, I don't even think to try it unless there's a corresponding control or menu item. It's also not universal, outlook is one of the most heavily used GUIs in the world and ctrl-f means forward in that. Then you've got the ubiquity of touch screens where ctrl-f isn't even an option.

> They were probably formally studying people who had never seen or used a GUI before

Everyone's seen a GUI, but that doesn't mean they've seen the particular GUI they're trying to use at any given time.

This leads to the Safari version Apple presented this week: even the "reload" button is gone now.

I think it is true that people learn and we can assume other things now. But still there must be discoverability of such features.

I believe there is space for great looking design that is also functional. The problem is that achieve this balance takes time and knowledge, and the software industry has a very short term focus, so you end up with non-solutions that are supported purely by hype.

Also, UI design has traditionally used visual styles from minimalism and the Bauhaus, mainly because the very low-resolutions where it was created (desktops first, later mobile screens) didn't support anything more ornate that would be still visually appealing.

I long for the day when more elaborate and luxurious art-deco-based interfaces become fashionable in UI design.

But minimalist design is not unique to software. If you look everywhere in our industrial society you'll see that. Even buildings follow this trend nowadays. It is a trend that derives from industrialization, where the goal is to make the simplest designs that still retain their utility.

I feel like every front-end engineer should be required to read The Design of Everyday Things and pass a quiz before being handed a job.

Building shitty UI because it looks cool in your marketing materials is doing a criminal disservice to your users and ultimately your business.

Most front end engineers don't get to decide how the UI looks, typically product and design will dictate that maybe with the exception of really small teams or products where design is apparently not priority.

Do you know of any good (presumably old) resources like books or articles that cover efficient economical GUIs and GUI frameworks?

Brad Myers. All the Widgets (Fixed v2) - 1990.

This was made in 1990, sponsored by the ACM CHI 1990 conference, to tell the history of widgets up until then. Previously published as: Brad A. Myers. All the Widgets. 2 hour, 15 min videotape. Technical Video Program of the SIGCHI'90 conference, Seattle, WA. April 1-4, 1990. SIGGRAPH Video Review, Issue 57. ISBN 0-89791-930-0.

https://www.youtube.com/watch?v=9qtd8Hc90Hw

Also by Brad Myers:

Taxonomies of Visual Programming (1990) [pdf] (cmu.edu)

https://news.ycombinator.com/item?id=26057530

https://www.cs.cmu.edu/~bam/papers/VLtax2-jvlc-1990.pdf

Updated version:

http://www.cs.cmu.edu/~bam/papers/chi86vltax.pdf

There were the original Apple and Gnome HIG (human interface guides). Also Ask Tog, and nngroup.com for web stuff.

Windows also had one.

And they are all ultimately derived from IBM CUA, so that's a good start: https://en.wikipedia.org/wiki/IBM_Common_User_Access

The classic Mac interface is a couple of years older, although most of these seminal works were published in the late 80s.

Ask Tog

https://www.asktog.com

What's up with the war on scrollbars!? I don't understand it.

the context of GUI use has also changed dramatically.. UIs were used to manipulate semi complex to complex formal documents in 2D, people were trained, it was all very important .. nowadays we can bring it down to swipe and tap over floating selfies. I'm sure nobody in the mainstream would ever use an 80s application no matter how good/better it is.

There's still plenty of office work to do. The selfies haven't displaced spreadsheets, they're a new category.

true but i have a feeling that for the next years to come, good old 2d static abstractions will be out of the picture (replaced by fully fluid animated UI paradigms)

People use Office and Adobe products. Sometimes you need a lot of functionality in your application which goes beyond simple swipe UIs.

>Is there a growing sense that our GUI/UI paradigms took a wrong step somewhere in CS history and we've pushed the current UI path as far as it will go?

Yes. It was called "web UI".

This brought a level of casualness and lack of rigor, that infected native UIs as well.

In the 90s there was a lot of GUI research, including by major companies like Microsoft and Apple, and lots of innovations.

But even more importantly, the standard practices of the main OS makers, favored a uniform look, with clear affordances (e.g. button bevels), and so on. The apex of with would be something like Windows 2000, Mac OS 8, BeOS, NeXT, and the like.

Stuff like "mystery meat navigation", "hamburger menus", "invisible scrollbars", huge padding, buttons that look like text, and so on, including "let's make the whole desktop app interface in the DOM" where post-2000 additions, and not for the better.

the latest version of xfce4 is what I would call a really clean, no bullshit modern desktop metaphor. I say that as a person who extensively used kde v1 and v2.x and migrated away from it when they started trying to clone windows vista.

In some ways xfce4 reminds me a lot of windows 2000.

Yeah, but unfortunately xfce4 just covers the desktop manager + some utility apps part.

Most of the apps would be an assortment of (non Xfce) Gnome-inspired GTK, Electron (e.g. Slack, VSCode), custom (Firefox, Chrome, Sublime Text), and/or KDE stuff, each with its own warts and "modern" approaches...

MATE would probably be closer in terms of its feature set.

Personally I think GUIs took a wrong step somewhere between the Alto and the first Macs. Specifically because we've lost this ability: https://youtu.be/AnrlSqtpOkw?t=605

People say that CLIs are composable and GUIs aren't, and so CLIs are better for power users. They're right, for today's GUIs, but yesterday's GUIs were composable. We've lost that.

We also lost this, https://www.youtube.com/watch?v=z_dt7NG38V4

Uhhhh... that demo is blowing my mind. Holy crap.

It's one of my absolute favorites. And don't forget, this predates the Macintosh.

Imagine where we could be today if this was our starting point in the 70s.

I think we just veered off the path a bit, and people like you are helping bring it back.

I myself am working on a web-based note-taking system which works with every browser since Mosaic. :)

> Is there a growing sense that our GUI/UI paradigms took a wrong step somewhere in CS history and we've pushed the current UI path as far as it will go?

Well, there are two conflated problems here:

1) The GUI and interacting with humans, itself.

I don't think we've even scratched the surface here. I'm not a big fan of the Instagram/TikTok "HAH! Everything is hidden, Boomer! If you aren't spending 5 hours a day with our application talking to your friends how to use our application you'll never figure it out." However, at least some stuff is different.

2) The software implementation of the GUI underneath

I do think we've gotten lost in a local extremum here. The whole "The GUI must run on the main thread" when we've got gajillions of cores spread across the CPU and GPU is a gigantic problem. We really need a multithreaded UI implementation.

To your point 1.

Younger generations don't feel like these features are hidden at all, and wouldn't feel like it's any different from most new apps. What Gen Z are very good at is discoverability inside apps using things like swiping. A younger person will interact with the app by swiping up and down or left or right, where older people might be more trained for button pressing. Having spoken to some younger people, they find some apps like Facebook harder to use, since there are too many buttons, and not enough contextual motion.

> Younger generations don't feel like these features are hidden at all

From watching the younger generation, I disagree.

There is no app discoverability or consistency at all. The first thing they do is go watch the video about how to do X in the app.

Sure, being younger, they can triple tap and multi-finger rotate better than older people. That doesn't make the GUI better.

I would love to see circular menus in GUI interaction on tablets. Sadly, only games seem to think they're a good idea.

Problem is that's hard to scale to sophisticated apps that typically run on the desktop, which are often used at work or even for school. Even if it's been ported to the browser, it usually more like FB than it is a messaging app.

And now that I think about it some more, I have a VR headset, and the UIs are often more complicated than swiping on your phone. It's more like pointing at buttons, menus and links with your controller. That's typically the case for gaming consoles as well.

As such, I'm not sure phone apps are still the latest in UI design, it's more that the smaller screen touch interface and apps that typically do one thing well better fit the form factor of smart devices. Granted, the younger generation use their phones for everything, until they put on a VR headset or have to work in a desktop-like program.

> The modern GUI, as we understand it, can be attributed almost entirely to the work of Douglas Engelbart.

Sketchpad by Ivan Sutherland was released in 1963, see e.g. https://en.wikipedia.org/wiki/Sketchpad. The NORAD/SAGE display systems which had light pens as pointing devices were even older.

There was no real concept of windows; the light pen was an input device for a small number of apps. I think the attribution to Engelbart is correct.

A "modern GUI, as we understand it" is one with a bitmapped screen, overlapping, non-modal windows and the desktop methaphor; why? because that's conceptually still what we have today; the first machine where this was realized was the Alto at Xerox PARC; the first applications we would consider "modern GUI applications" were the Bravo WYSIWIG text editor (1974), the Gypsy document preparation system (the first existing DTP application, 1975) and the Smalltalk-76 environment (whereas overlapping windows and pop-up menus were already present in Smalltalk-74). Engelbart's and the other projects mentioned have undoubtedly made important contributions to the first graphical user interface; however, there were still various hurdles to overcome before the first "modern GUI" came into being.

I don’t remember using any overlapping windows in the Alto environment though the display was bitmapped; Bravo, for example, was a “full screen” program. This wasn’t that odd; for example the CADR environment was also non-overlapping-window system. The iPhone was the modern reincarnation of this.

The earliest “overlapping windows” system was, iirc, the smalltalk environment.

I say “environment” because that back then you booted the machines (Alto and later the D machines) into a completely hermetic environment; Smalltalk, Interlisp-D, and Cedar/Mesa ran on the bare hardware (had their own networking stacks and drivers and even had their own microcode. There was no meaningful distinction between OS and application.

My memory is a little fuzzy as this was quite a while ago and there was a lot of experimentation going on all over the place. I left PARC to go back to MIT in 1985.

Bravo was modal, Gypsy was non-modal, Smalltalk-74 onwards had multi-windows (including pop-up menus) and a metaphorical desktop.

I assume you know these interesting presentations: https://www.youtube.com/watch?v=2Z43y94Dfzk, https://www.youtube.com/watch?v=uknEhXyZgsg

When I get asked to give talks about interaction design I show Figure 3 from Kay and Goldberg's Personal Dynamic Media (1977) [1], with the "interim Dynabook" part of the caption redacted, and ask the students to fill in the blank. They usually guess "personal computer" and are then shocked to learn the actual framing and the overall backstory.

[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.656...

I'm not sure I understand. What would you expect them to guess? What are they shocked at?

Amiga, GEOS, GEM. If you want to talk about the early GUIs you can start with Xerox I think, but Apple was not the only one back then.

https://en.wikipedia.org/wiki/History_of_the_graphical_user_...

Indeed, the genesis of GEM was with Lee Jay Lorenzen who left Xerox to come to DR after he tried to pitch a light version of the Star's GUI that would run on PC-class hardware. So although there was obviously a lot copied from the Mac there was also a lot that came from his experience at Xerox.

Here's his presentation of his prototype, in 1982, quite a while before the Mac or Lisa were released:

https://www.youtube.com/watch?v=EMBGRZftS30

> Most amusingly, Engelbart found it unlikely that computer users would want to use a mouse with one hand when the keyboard requires both.

This has, in practice, proven to be basically wrong, yes.

But I giggled reading it because I'm absolutely addicted to trackpoint based keyboards specifically to avoid needing to move my hands (for both my convenience and the sake of my wrists).

Edit: ... and now I note that there's a footnote saying the author shares said preference.

> The thing is, CP/M and DOS were both primitive operating systems by modern standards. CP/M and DOS were not multi-tasking.

One of the reasons is that the first Intel processor to support multi-tasking was 386. When DOS was created, there was no hardware support for multiprocessing and virtual memory.

A 32-bit CPU with MMU is not a prerequisite for multitasking. The Amiga (and others) managed preemptive multitasking on a Motorola 68000.

For designs that wanted an MMU, you could use an external one. Sun had external MMUs in their early workstations. Even Apple had their own MMU in the Macintosh II line before they switched over to the 68030.

> Even Apple had their own MMU in the Macintosh II line before they switched over to the 68030.

You could add a 68851 to a Macintosh II, I didn't think Apple developed their own MMU for it. The Lisa had a custom MMU.

Ah, I misremembered. Wikipedia's Macintosh II page says:

"The machine shipped with a socket for an MMU, but the 'Apple HMMU Chip' (VLSI VI475 chip) was installed that did not implement virtual memory (instead, it translated 24-bit addresses to 32-bit addresses for the Mac OS, which would not be 32-bit clean until System 7)."

Minix, PC/IX and Microsoft/SCO Xenix were indeed multitasking systems (well, Unices) running on the 8086

> One of the reasons is that the first Intel processor to support multi-tasking was 386.

Not quite. The 80286 was not a good CPU, but it could do multitasking:

https://www.cpu-world.com/CPUs/80286/index.html

> The second generation of x86 16-bit processors, Intel 80286, was released in 1982. The major new feature of the 80286 microprocessor was protected mode. When switched to this mode, the CPU could address up to 16 MB of operating memory (previous generation of 8086/8088 microprocessors was limited to 1 MB). In the protected mode it was possible to protect memory and other system resources from user programs - this feature was necessary for real program multitasking. There were many operating systems that utilized the 80286 protected mode: OS/2 1.x, Venix, SCO Xenix 286, and others. While this mode was useful for multitasking operating systems, it was of limited use for systems that required execution of existing x86 programs. The protected mode couldn't run multiple virtual 8086 programs, and had other limitations as well:

> 80286 was a 16-bit microprocessor. Although in protected mode the CPU could address up to 16 MB of memory, this was implemented using memory segments. Maximum size of memory segment was still 64 KB.

> There was no fast and reliable way to switch back to real mode from protected mode.

More about OS/2 on the 80286:

https://www.landley.net/history/mirror/os2/history/os210/ind...

Your quote states the problem of the 286: it was cumbersome to use and had many limitations that were solved only with the 386, which was the first successful Intel processor to implement multitasking and virtual memory.

Good but tantalizing reference to the document-oriented vs program-oriented interface. I thought I was the only one who still carried a torch for a document-oriented model. Anyone who first used a personal computer after, say, the mid 1980s probably has no idea what they are missing.

What am I missing?

Note, the previous post referenced (but not linked) in the text appears to be this: https://computer.rip/2021-05-24-dialogs-not-taken.html

If you want to see GUI evolution and the current state of the art, you have to look at very specific video games. Namely, ones where modding is allowed and acting smarter or faster is a competitive advantage.

From the article:

> Despite the best efforts of many dweebs, one handed text entry has never caught on [2].

One-handed text entry may not have, but putting one hand (usually your left) on the keyboard and one on the mouse, and using your voice for text related communication has taken over PC gaming. I wonder what an equivalent productivity app (or set of emacs modes) would look like.

Obligatory reference to the mother of all demos [1] by Douglas Engelbart

[1] https://www.youtube.com/watch?v=yJDv-zdhzMY&ab_channel=Marce...

I wonder when we'll see the last GUIs, and everything will be speech-based.

> I wonder when we'll see the last GUIs, and everything will be speech-based

The answer is never. The reason is that different kinds of UIs are effective for different kinds of tasks.

I can't remember who I heard this from, but they asked the audience to imagine a speech-based steering wheel for cars. "Turn left... more... more... MORE!!! LESS! LESS!"

Speech also has problems with sensitive data in public places ("Please enter in your password" or "Your stock portfolio is now worth ..."), has problems with noise, and can also interfere with some parts of your cognitive processes. In addition, speech is serial, which means that you can pretty much only process one stream at a time. Contrast this with a web page that you can skim, or a visualization that can show a lot of data.

Different kinds of interactions have different tradeoffs. Speech is good for some tasks, and not very good for others.

> I can't remember who I heard this from, but they asked the audience to imagine a speech-based steering wheel for cars. "Turn left... more... more... MORE!!! LESS! LESS!"

Star Trek fans probably would know the correct answer: don’t use a steering wheel; that’s just a means towards a goal, not the goal itself. Everybody who ever used a taxi could have figured out the proper speech UI for a car, too: “Take me to the airport”/“follow that car”/etc.

We don’t have touch-based ovens where you prod the fire with your bare fingers or cars where you have to tap a screen to inject gasoline at just the right time, either.

Actually, the ideal UI is like a good butler: you don’t even have to tell him what you want; he just knows.

But yes, the answer likely will be ‘never’.

Also, human vision is the single most important way we understand the world, the visual cortex is also proportionally much larger than other sensory organs’. It would be a downgrade to restrict our communication with computers to a more narrow channel.

Definitely agree with never, but I'll let Bret do the talking!

http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...

I've always thought the last interface would effectively be notepad/text.

As long as there are minds there will be text.

As long as text can be complex we will have to consume it in blocks and probably visually (or the direct brain interface equivalent)

I like where you are headed with this. I’ve put some thought into this recently and I’ll post the condensed form.

I believe you are correct, in a sense, that as long as there are minds there will be text. However, I believe it should be considered from a slightly different angle. As long as there are minds there will be language. Text is one representation of that and it is a representation we can manipulate easily computers. This makes text a powerful and simple medium that can be used pretty much everywhere (f.g. Unix-like systems; networking protocols; data storage).

Language is universal. Intelligent entities will create and use language to communicate with other entities or with themselves (i.e. sticky note). We have been proven capable of learning language even in some of the most input restrictive states, such as the case of Helen Keller. Language can be seen, heard, or felt. I wouldn’t be surprised if some creatives come up with a way to smell or taste it. Computers can be read, heard, or felt (f.g. Braille terminals). Text is a useful medium for storing the basic information that is then transformed into the output most suitable for the final representation given to the end user.

Let me bring this back around to your post. Because of the aforementioned reasoning, while I generally agree with your post I do disagree that text must be consumed visually or in blocks. The information contained in text should be transmitted in whatever way fits the user best. I realize this is vague, but I think this a starting point to thinking further on how to best get information to the user. Personally, I've come to the conclusion that a flexible core application should send and receive inputs and outputs to a seperate interface application. This may take the form of a typical client-server style system. This way, the interface can be more easily to the user's interfacing preferences and the core application only concerns itself with its task. Large blocks of text may not be well suited in certain instances, such as the case with screen readers or Braille terminals.

I'm trying not to be too general or vague here. I don't feel I've presented enough to really provide any actionable insight here. But I see this post is getting long, and I will leave it at this for now as potential food for thought for yourself or others. It's a concept I'm still thinking about how to best implement, and the details of implementation across various OS's and devices complicate matters.

You may want to check this beautiful (yet expensive) beast ;-)

https://remarkable.com/

I don't think it is that expensive for what it is.

Neither do I, I bought one ;-) But people comparing it to Kindle thought it should be about 200€ tops, because of its limited function compared to a tablet.

I don't know that speech will kill GUIs (your eyes allow much higher bandwidth than your ears), but direct neural interfaces might.

Never, because speech is a terribly ambiguous and low-bandwidth communication technique to transfer technical information and commands. Not to mention that it is terribly inconvenient if several people in the same room use it at the same time.

I dread that moment, if it ever happens. Imagine being on the subway or in a cafe with a dozen people talking to their devices. And say goodbye to privacy; everyone will know what you're doing.

Considering the number of people I've seen on buses and trains shouting "OP-ER-AY-TOR!" into an IVR system, we're already halfway there.

I wonder when we'll see the last GUIs, and everything will be speech-based.

Not in our lifetimes.

People have been promising speech recognition since at least the late 70's. Today's "digital assistants" are little more than parlor tricks, on those occasions when they work.

They don't seem much more accurate than my old Covox Voicemaster.

They are quite accurate. But most things we do on computers are things we want to look up and see. Text is scannable, a voice message is not. You can pan and zoom in a map, but not in dictated instructions. Listening to books and articles is fine, but you want to see a movie.

Reply on news.ycombinator.com