Noise cancellation improves turn-taking for AI Voice Agents
krisp.ai
krisp.ai
Does anyone know an alternative that achieves similar level of background noise cancelling?
Since then we've been relying solely on Zoom's noise cancelling features and haven't been missing anything. They really improved massively over the years.
While it's not a drop in solution if you routinely need to join calls with other conferencing software, I haven't missed Krisp's after 'switching' 2 years ago.
If you're on Windows, NVIDIA Broadcast's AI Noise Removal has performed similarily well for me while gaming.
I will look again at zoom, last time I tried their noise canceling it would cut my sentences
https://support.apple.com/guide/mac-help/use-mic-modes-on-yo...
The app now really wants you to use their task tracking, transcripts, summaries, etc. But they have pretty frustrating UX, require invasive permissions. Every other day you get nagged into enabling features you already said no to 20 times
When a human speaks to another, the second person will listen and interpret and guess when the first person is finished talking. For voice agents it doesn't work that way at all.
The text-to-speech system just seems to have a hardcoded "pause" detector, e.g. 2 seconds, and if 2 seconds of silence are ever detected, the "end of message" token is sent and the LLM will start talking. Even if you were just collecting your thoughts and weren't finished at all.
So the semantic content of what you are saying is completely ignored for turn-taking and no analysis takes place which would determine whether the user is likely to have said everything they wanted to say.
Instead of the rigid pause detector, it would actually make more sense for the end-of-message token to be sent when you explicitly say a specific phrase, like literally "over". Which was of course common in half-duplex radio where only one person could transmit. LLMs are half-duplex too: they can't listen and talk at the same time.
That doesn’t sound very conversational at all. Instead one could train the network to recognise the appropriate turn-taking points.
The simple way to do that is to make the model output a “listen a bit more” token when it is not yet their turn to talk. You can use real life recorded conversations to build up the initial training set, and then add more data where clashes happen (where tha AI and the speaker speaks at the same time over each other.)
More complicated would be a system where the model is periodically fed the audio chunk so far, and the model predicts what the speaker is likely going to say and based on that when it is appropriate to respond and with wath. And then a smaller, faster, local model can be used to verify if what was said matches the prediction, and if so outputs the generated response. If there is a mismatch it engages the more expensive model to come up with a new prediction.
If you engineer this right you can reuse the state vector from save points and save a bit of compute that way.
Asking the user to say “over” at the end of their turn is the most heavy handed solution. Recognising the flow of a conversation is just pattern recognition. That is what machine learning is good at.
And surprisingly hard to do well in practice. My guess is that the problem is that there is very little information in your training dataset (because only the transition from "talking" to "done talking" matters), but the actual knowledge required to perform well is large (up to and including full speech recognition, in theory). So even with over a terabyte of training data, your choices are a small model that performs badly or a large(r) model that overfits severely.
It's possible there was something I was overlooking when I tried it, though. I couldn't think of a good way to confirm my guess experimentally.
https://platform.openai.com/docs/guides/realtime-transcripti...
I guess overall latency will be higher since processing will have to go to their server than back to our server then back from our server to STT provider and back to as then back to LLM provider and back to us and last part to TTS provider and back to the user.
It's so weird that e.g. OpenAI doesn't provide a way to make very simple voice pipeline STT + LLM + TTS executed totally on their servers, this would reduce latency significantly.
The pipeline with this server side audio processing right now looks mostly would have to look like that:
user phone -> our server -> krisp server -> our server -> OpenAI STT -> our server -> OpenAI LLM -> our server -> OpenAI TTS -> our server -> back to user.
Then you have to hope that user and all servers are hosten in the same region
I’m looking forward to more UX improvements in voice pipelines now that the big players have stabilized their voice pipeline frameworks
I wrote a "simple" (ugly) Acoustic Echo Cancellation module that kind of worked, but wondering if anyone had any solutions to make it work over the WebSockets Realtime API
My own system automatically detects new speakers and tries to pick up on cues to identify the speaker, and once they are identified by name, the corresponding average embedding is inserted into a vector database so that the agent can later use the embedding for simple authentication, ignoring chatter in noisy public spaces, RAG context loading, etc. It works pretty well!
Background noise will rarely produce a false positive for that word.
As a result, the VAD mistakenly interprets noise or background voices as active user speech, triggering unintended interruptions. These false triggers negatively impact turn-taking, a core component of natural, human-like conversational interactions.