Generative AI is also a revolution for computer interfaces | Hacker News Reader

Generative AI is also a revolution for computer interfaces

axios.com

84 pointslegrande50 comments

> Think of the hours saved — and, in theory, productivity gained — if you can simply tell your chatbot to clean up your inbox, change your system settings or connect to a printer.

I have a difficult time imagining myself trusting something as unpredictable as an LLM to perform any kind of "cleaning up" task on important data like my inbox. Perhaps future developments in utilizing language models in conjunction with less stochastic systems will make this an easier sell, but for now, I'm not seeing it.

As someone who's been working on this space as their day job for the last several months, I can say that LLMs are SURPRISINGLY GOOD at this. The real challenge is the sentiment you're articulating here...trust. I've been relating it to self-driving cars. I love the idea, but I'm going to be very anxious the first several times I climb into one to take me somewhere at freeway speeds.

The biggest UX work in the short term is going to be around giving people the right amount of transparency so they still feel in control. The actual LLM work is relatively easy and impressive.

Haven't they recently discovered that Tesla crashes are far more frequent that previous estimations?

Whatever that number is, how does it compare to crashes based on human error?

Tesla won't release the numbers so we don't know.

It’s also weird to compare the Tesla to some ‘99 Toyota. Why not compare across the pool of cars that also have adaptive cruise control, lane keep assist, forward collision prevention, etc..?

I recall the number of deaths from Tesla self-driving being reported as being 10x per mile as regular highway driving. I think it was this article, but it's paywalled. https://www.nytimes.com/2022/06/08/technology/tesla-autopilo...

Similar to self-driving cars, I think the last 1% of the way is going to be the near-insurmountable barrier that makes widespread implementation difficult or potentially impossible. For tasks where you can plausibly check the results with some kind of external software harness, or defer double-checking to a person, these models will make development quick (if resource-intensive). But any kind of fully-automated complex process where the consequences are important will be tough to hand over to black-box generators.

But is surprisingly good useful?

Suppose you configure your inbox such that you unpredictably don't see a small percentage of important emails. Suppose you miss an email from your boss telling you a very important client needs you to fix a critical bug today, and as a result you start working on that bug slightly later when your boss contacts you another way. Your boss might tell you to uninstall your LLM and just read all your email

The same logic could be applicable to pretty much any other spam filtering mechanism.

No, it's the same category, but the specific values makes all the difference. The logic depends on the probability of error, the predictability of error, and the social acceptability of error.

By analogy it's socially more acceptable to have an outage because AWS is down than because insert_nich_provider is down even if insert_nich_provider has fewer outages. If insert_nich_provider has more outages it's even worse.

How can you get transparency from an LLM? They're complete black boxes by architecture.

For inference? Not really. LLM observability startups are a thing. Arize.ai probably the most well known.

>How can you get transparency from an LLM? They're complete black boxes by architecture.

You just ask them. I literally just prompted ChatGPT (3.5) to think about its answer again and explain what's wrong with it, and it did. Kept going 2 or 3 times, until it was sure it wasn't doing anything wrong, and magically the script it produced at that stage just worked.

But can you believe that these are the correct explanations? Perhaps it just confabulated an explanation that sounds likely given your prompt, and is not the actual reason it told you what it did?

That'll just give you made up rationalizations, like asking a human to explain an emotional subconscious choice.

I know from use that perplexity.ai (which builds atop GPT4) hallucinates very little compared to Bard and ChatGPT.

What are some techniques that these products use apart from bounding context and prompt engineering? ReAct sounds like it might help: https://react-lm.github.io/

I believe you. I think for at least some people this might be one of those generational differences.

Wanting to do some things by hand might be increasingly a thing only people over a certain age do.

Yeah this is what I call the Minority Report fallacy. People see or imagine an interface that looks or sounds cool, but don't think through what it would be like to really use it every day.

Do I really want to wave my arms to control an interface? No, they'd get tired immediately. A mouse on a desk where I can rest my arms isn't sexy because it's not new, but it's pretty much ideal.

"Do I really want to wave my arms to control an interface? No, they'd get tired immediately."

They would, but only because you're not used to using them much.

Using our arms more would probably be really good for us.

If we had to use our legs and feet to move a mouse then computer programmers might be some of the fittest people on the planet, instead of some of the fattest.

I say this while lying on my bed and tapping with my fingers on my tablet... we're doomed...

You're right, we should probably all work out more, but gravity is gravity and I'd still rather not fight it all day.

If you consider assembly line or like, Amazon box packing workers, they move their arms all day already, fighting gravity

And. I'm sure most people here don't want to be an Amazon box packing worker even if the pay is comparable to programmer's.

You would trust it by having it work in steps.

Convert your command into the correct syntax.

Preview what the execution of the command looks like.

Review the changes after they occur.

Undo.

What if an LLM provides socially acceptable plausible deniability?

"Sorry I didn't see your important email; my AI assistant deleted it"

I entirely dislike this interface paradigm, but I don't think that is any more of a problem than "weird... it got sent to my spam folder."

I've had great luck with chatGPT assisted FFMPEG commands.

There is very little that FFMPEG can't do as far as video manipulation, but getting a good CLI invocation is a lot of work, documentation surfing, and piecing things together from stackoverflow and Reddit. I don't really feel like committing such things to memory.

With ChatGPT, I can ask for it to make a parameterized bash script that takes a path for video input and output, and any other parameters of consequence and make a "converter from x format to Y using Z codec and nvidia hw accelerated encoding at W bit rate" and most of the time, the bash script that it gives me just works and I move it into my tool-bin.

It's nice! There are probably a lot of other tedious pieces of automation that it can handle well even in its early state.

I've found ChatGPT very useful for tools that I use once in a very long while (like ffmpeg) and simply don't remember all the nuances of.

For example, I needed to use K6 to run a load test. It supports a million things, and it would have taken me a long while to figure out the right combo. With ChatGPT I had it running in 5 minutes.

What is your process of verifying it?

You run it. If it gives an error or the output isn't what you want, you tell Chatgpt to try again.

Also using 4 and not 3.5.

I imagine there is so much pointless discussion online with one person talking about 3.5 and the other person talking about 4.

I think a good way to verify that the commands are actually doing something instead of the LLM just generating non-existent parameters is to provide a dry run mode. The actual correctness of the generated command - granted it's valid - performing as it should however, is a whole different issue.

I also have had much success with FFMPEG and other bash commands - I just ask it to explain what the command does step by step as I want to learn a bit as well.

That being said - it is far from perfect.

I don't think so. Chat based interface is horrible on phone. You have to type what you want instead of tapping things.

Chatbot should be part of your ui instead of replacing the ui.

I hear this often, but you don't need to surface the output of an LLM as chat

I made https://notionsmith.ai and technically it's "chat based", but the user isn't exposed to 99% of the chatting: the only chat interface on the site is there more as a bonus feature than anything.

I agree your app shouldn’t be labeled 100% a chat bot, but the primary method of interacting with it is via natural language, which is the fundamental difference the article is describing.

> You have to type what you want instead of tapping things.

On phones, I'd expect that voice will be a more popular input method than virtual keyboards.

It’s not really convenient in many situations.

That works in isolation, but not when you're not home. Do you want to talk to your phone in a public restroom, or the subway, or in the office?

From what I've observed, people have no problem talking to other people in public places. I don't see why this behavior would change for non-human listeners.

I regularly use speech-to-text transcription publicly. With my mouth close to the mic, I can speak softly enough that people outside of my personal space can't hear.

That being said, keyboards still have their place. Certainly, I use ChatGPT on my phone several times a week using keyboard input, and it's not particularly onerous.

Personally I’m mildly disturbed at the legion of people currently staring at and tapping at their phones suddenly switching to muttering at them.

People do that already. It's just that when they do so it's normally a phone call.

There's no reason people couldn't make interfaces that use the same speaker as a phone call does.

Once your chat voice listener can access your private information like bank accounts, etc., I imagine things will listen to you everywhere. Perhaps every phone or device will be listening like Find My Devices except being Take My Money.

Maybe instead of a keyboard of letters there is an array of words to build commands from.

This doesn’t have to be for complex or really critical tasks to have value.

Just having it able to work a smart home or do basic things while talking natural is a big deal and a big increase in useful interface.

I really like my Alexa - I love the _idea_ of a conversation interface, except an Alexa isn’t that. It’s a call and response only. I’d love to be able to talk to a computer out loud conversationally that can take actions / give me info.

Even convos as simple as

‘Hey what’s the weather tomorrow’

‘Tomorrow will be 25c and raining’

‘What time will it rain?’

‘Not until 4pm’

This simple stuff is currently WAY outside the abilities of any ‘smart’ assistant.

You can get that today with ChatGPT plugins aside from the voice part. But I assume OpenAssistant can do it now if it's properly configured.

Certainly with a weather API and the new ChatGPT API functions thing you can build that in a few days or less. Combine it with Whisper or Google voice input and Eleven Labs very realistic speech output.

I think there are quite a few groups with demos that can handle that particular weather thing and basics like it now.

Actually Google Assistant probably can handle follow ups to some degree.

Would first need to let chatGPT do a separate call to figure out what API to invoke.

I think generative AI will eventually see very wide application on this area and it might be one of the most magical uses for it.

Btw, I think Tim Cook should have released Vision with thought controls. Then it'd have been revolutionary....

Can't wait to have every UI make me wonder why I can't GET YE FLASK

> what is discoverability

Reply on news.ycombinator.com