Hacker News Reader

Comment by lucubratory | Hacker News Reader

How hard on your end does the task of making the chatbot converse naturally look? Specifically I'm thinking about interruptions, if it's talking too long I would like to be able to start talking and interrupt it like in a normal conversation, or if I'm saying something it could quickly interject something. Once you've got the extremely high speed, theoretically faster than real time, you can start doing that stuff right?

There is another thing remaining after that for fully natural conversation, which is making the AI context aware like a human would be. Basically giving it eyes so it can see your face and judge body language to know if it's talking too long and needs to be more brief, the same way a human talks.

View on HN

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted.

The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is ready it starts over. If it finishes its response before any more words arrive (~1 second usually) it starts speaking. This is not ideal because the user might not be done speaking, of course. If the user continues speaking the bot will stop and listen. But deciding when the user is done speaking, or if the bot should interrupt before the user is done, is a hard problem. It could possibly be done zero-shot using prompting of a LLM but you'd want a GPT-4 level LLM to do a good job and GPT-4 is too slow for instant response right now. A better idea would be to train a dedicated turn-taking model that directly predicts who should speak next in conversations. I haven't thought much about how to source a dataset and train a model for that yet.

Ultimately the end state of this type of system is a complete end-to-end audio-to-audio language model. There should be only one model, it should take audio directly as input and produce audio directly as output. I believe that having TTS and voice recognition and language modeling all as separate systems will not get us to 100% natural human conversation. I think that such a system would be within reach of today's hardware too, all you need is the right training dataset/procedure and some architecture bits to make it efficient.

As for giving the model eyes, actually there are already open source vision-language models that could be used for this today! I'd love to implement one in my chatbot. It probably wouldn't have social intelligence to read body language yet, but it could definitely answer questions about things you present to the webcam, read text, maybe even look at your computer screen and have conversations about what's on your screen. The latter could potentially be very useful, the endgame there is like GitHub Copilot for everything you do on your computer, not just typing code.

you'd have to do something along the lines of what voice comm does to combat the output feedback problem. i think it involves an fft to analyse the two signals and cancel out the feedback, im not 100% sure on the details.

I plan to change the audio input to use WebRTC, then I get echo cancellation and network transparency for free. Although dealing with WebRTC is a headache harder than doing the AI parts.

What do you find hard about WebRTC?

I would love to help. Would even code up a prototype if you wanted :)

For starters, every WebRTC demo I've tried has at least 400ms of round trip latency even on a loopback connection. Shoot me an email if you know WebRTC, would be good to chat with someone who knows stuff!

Thanks, fascinating insights. I think an everything-to-everything multimodal model could work if it's big enough because of transfer learning (but then there are latency issues), and so could a refined system built on LLMs/LMMs with TTS (like what you are using), but I haven't seen any good research on audio-to-audio language models. My suspicion is that that would take a lot of compute, much more than text, and that the amount of semantically meaningful accessible data might be much lower as well. And if you do manage to get to the same level of quality as text, what is latency like then? Not 100% sure, just intuitions, but I doubt it's great.

I like the idea of an RL predictor for interruption timing, although I think it might struggle with factual-correction interruptions. It could be a good way to make a very fast system, and if latency on the rest of the system is low enough you could probably start slipping in your "Of course", "Yeah, I agree", and "It was in March, but yeah" for truly natural speech. If latency is low you could just use the RL system to find opportunities to interrupt, give them to the LLM/LMM, and it decides how to interrupt, all the way from "mhm", to "Yep, sounds good to me", to "Not quite, it was the 3rd entry, but yeah otherwise it makes sense", to "Actually can I quickly jump on that? I just wanted to quickly [make a point]/[ask a question] about [some thing that requires exploration before the conversation continues]".

Tuning a system like this would be the most annoying activity in human history, but something like this has to be achieved for truly natural conversation so we gotta do it lol.

> although it does require you to wear headphones so the bot doesn't hear itself and get interrupted.

Maybe you can use some sort of speaker identification to sort this out?

https://github.com/openai/whisper/discussions/264

Yes, this is a good idea. Too many good ideas, not enough time!

A simple correlation of audio chunks from microphone and from the TTS should be enough to tell which parts in the input stream are re-recorded TTS. Much simpler, no?

It's not so simple when the impulse response of the room and mic and speakers are all unknown, possibly changing, plus unknown background sounds as well, possibly at a very high level. and there's also unknown latency which can be quite large especially in the networked case, and maybe some codecs, and maybe some audio "enhancement" software the OEM installed on the user's machine. Also, ideally the computer would be able to hear the user even while it is speaking.

Echo cancellation is non-trivial for sure.

Could it be done more reasonably with the transcription? With diarisation the logic of "someone is saying exactly / almost exactly what I'm saying, that's probably me" might be pretty reasonable.

Short-term could it be configured as push to talk?

Certainly, but then it has little advantage over e.g. ChatGPT voice mode. I guess running locally is an advantage but the voice and answer quality is worse. The much better latency and more natural conversation is what I like about it.

Am I wrong to think it would have a couple major advantages? Like using speakers without having to worry about echo cancellation, having a distinct interrupt signal, and still getting all the latency benefits (possibly even more once you get used to it since the conversational style has to assume the end of the user’s sentence instead of knowing the second they let go of the button)

You wouldn't quite have all the latency benefits because you'd have the additional delay between when you stop speaking and when you release the button (or cut off speech if you release too early). It wouldn't respond any faster because it's already responding at the fastest possible speed right now, it doesn't wait at all. And it wouldn't be hands free, it wouldn't feel like a natural conversation which is what I'm going for.

I'd rather use speaker diarization and/or echo cancellation to solve the problem without needing the user to press any buttons.

What I would really like is a push-to-talk app on my phone, so I can talk to my house from anywhere without worrying about putting a microphone in every room, or monkeying about with wakeword detection. I'm sure it's doable, I'm just not conscious of having seen it done.

Deciding when someone is done speaking is hard to do well and impossible to do perfectly. Some people finish speaking, then think of something else to say and pretend they were still talking.

True, perfection isn't achievable but human level performance is all you need and it may be possible to do better than that.

Instead of sacrificing flexibility by building one monolith model that does Audio to audio in one go, wouldn't it be better to train a model that handles conversing with the user (knows when the user is done talking, when it's hearing itself, etc) and leave the thinking to other, more generic models?

You don't lose flexibility with an end to end model. You lose controllability. But there are ways to mitigate that.

> It could possibly be done zero-shot using prompting of a LLM

That's how I've been thinking of doing it - seemed like you could use a much smaller GPT-J-ish model for that, and measure the relative probability of 'yes' vs 'no' tokens in response to a question like 'is the user done talking'. Seemed like even that would be orders of magnitude better than just waiting for silence.

It would be very interesting to have something like BakLLaVA's image description fed from a webcam used as a context for the LLM. "You can see: <description of scene, description of changes from last snapshot>" or something along those lines in the system prompt.

I installed this on my system, having a little issue like you mention below where the bot hears itself so it got into a loop of talking to itself and reply to itself.

Could the issue be that I am using a pair of bluetooth headphones and the microphone is built into that - what is the optimum setup, should I be listening on the headphones and using a different mic input instead of the headphone mic?

This is pretty intermeeting I would love to get it to work. Running a 3060.

Thanks.

I installed this on my system, having a little issue like you mention below where the bot hears itself so it got into a loop of talking to itself and replying to itself, it was pretty funny.

This is pretty intermeeting I would love to get it to work. Running a 3060.

Thanks.

That's weird, Bluetooth headphones should work I think. Maybe there's something wrong with them. You can check if the mic can hear the speakers using Windows sound recorder. Try a different mic if you have one, the only important thing is the mic can't hear the headphones.

Reply on news.ycombinator.com