Hacker News Reader

Comment by localhost | Hacker News Reader

There are two things that I think are needed and that I'm not sure if anyone provides yet to make this scenario work well:

1. Interruption - I need to be able to say "hang on" and have the LLM pause. 2. Wait for a specific cue before responding. I like "What do you think?"

That + low latency are crucial. It needs to feel like talking to another person.

View on HN

> Interruption

Well, today is your lucky day!: https://persona-webapp-beta.vercel.app/ and the demo https://smarterchild.chat/

The latency on this (or lack thereof) is the best I've seen, would love to know more about how it's achieved. I asked the bot and it claimed you're using Google's speech recognition, which I know supports streaming, but this result seems much lower lag than I remember Google's stuff being capable of

> I asked the bot and it claimed you're using Google's speech recognition

That doesn't sound plausible. How can the LLM part know which speech recognition service is being used?

It's not entirely unlikely that the llm is informed exactly what it's source data is, with the hope that it can potentially make corrections to transcription errors

Or just because it's interesting and people might ask. I could imagine it being a hallucination, but it could also be an easter egg of sorts.

Apparently it uses the Web Speech API [1], not a specific service.

[1] https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecog...

this crops up in my feed every now and then and it has vastly superior perf vs. ØAI’s ChatGPT iOS app or anything else I’ve found. truly outstanding. are you planning on developing it further and/or monetizing it?

This isn't mine, it's from sindarin.tech, they already have paid versions, with one plan being $450/50 hours of speech (just checked and it's up from 30 hours).

The latency on smarterchild is very fast, but it doesn't seem to be interruptible. The UI seems to be restricting me from even inputting input in between my input and the ai response?

I had no problem with “hold on a sec” and then “sorry, please continue”

I didn't think low-latency high-quality voice chat would make such a difference over our current ChatGPT chat, but oh my, I think that really takes it to the next level. It's entering creepy territory, at least for me.

I clicked on "Talk", but the textbox just says "Preparing to speak..." without doing anything else

It doesn't work for me on Firefox, but works on Vivaldi.

In order to feel like a human, cues should not be a pre-programmed phrase, the system should continuously listen to the conversation, and evaluate constantly if speaking is pertinent at that particular moment. Humans will cut a conversation if it's important, and such a system should be able to do the same.

Totally agree with your take. But a pre-programmed phrase would work today and hopefully wouldn't be too difficult to implement. I would imagine that higher latency would be more tolerable as well. But in the fullness of time, your approach is better.

When I'm listening to someone else talk, I'm already formulating responses or at least an outline of responses in my head. If the LLM could do a progressive summarization of the conversation in real-time as part of its context this would be super cool as well. It could also interrupt you if the LLM self-reflects on the summary and realizes that now would be a good time to interrupt.

Indeed a great point. Waiting for a specific cue, before responding, is an interesting idea. It would make the interaction more natural, especially in situations where the user is thinking aloud or formulating their thoughts before seeking the AI's input.

Interruption is something that is already in the pipeline and we are working on it. You should see an update soon.

Thanks! Really looking forward to interruptions.

I think about the cue as kind of being like "Hey Siri/Alexa/Cortana" but in reverse.

I agree, it is unnatural and a little stressful with current implementations. It feels like I first need to figure out what to say and than say it so I don’t pause and mess up my input.

I hope the new improved Siri and Google assistant will be able to chain actions as well. “Ok Google, turn off the lights. Ok Google, stop music.” Feels a bit cumbersome.

A fast turnaround time is also super important; if the transcription is not correct, waiting multiple seconds for each turn would kill the application. E.g., ordering food using voice is only convenient if it gets me right all the time; if not, I will fall back to the app.

I wrote a sort of toy version of this a little while ago using Vosk and a variety of TTS engines, and the solution that worked mostly-well was to have a buffer that waited for audio that filled until a pause of so many seconds, then it sent that to the LLM.

With the implementation of tools for GPT, I could see a way to having the model check if it thinks it received a complete thought, and if it didn't, send back a signal to keep appending to the buffer until the next long pause. The addition of a longer "pregnant pause" timeout could have the model check in to see if you're done talking or whatever.

To streamline the experience we don't send the transcription to the LLM after the pause, since we are using the time we wait for the end of sentence trigger (pause) to generate the LLM and text-to-speech output. So ideally once we detected the pause, we already processed everything.

I agree. Have been working on a 2 way interruptions system + streaming like this. It's not robust yet, but when it works it does feel magical.

> 2. Wait for a specific cue before responding. I like "What do you think?"

"Over."

"Over and out" closes the app. ;)

Saying "Go" to indicate it's the bot's turn would work for me. (Or maybe pressing a button.) The bot should always stop wherever I start speaking.

> I like "What do you think?"

I like it too!

And I can't help but think getting into the habit of saying it -- it would help us get along much better with other people in our lives.

I did a video demo of this. Tell it to only respond only with OK to every message and only respond fully when I tell you I am finished. Ok? Ok.

It would be cool if the Ai could interrupt too.

"Imma let you finish, but..."

Reply on news.ycombinator.com