davitb

This is for listener-side, not speaker-side. So no misuse case here.

A small, audio-only turn-taking model from Krisp

They recently added noise cancellation to realtime transcription.

https://platform.openai.com/docs/guides/realtime-transcripti...

The hard part is to separate background voices (e.g. TV, chatter, etc) from the primary speaker's voice. Basically do voice isolation. Voice fingerprinting would help only in this context.

On average, an hour of speech contains about 9,000 to 15,000 words. This range accounts for different speaking speeds, which typically vary from 150 to 250 words per minute.

So this translates to tens of millions of words.

In the last 3 months our team worked hard to build Krisp Chrome Extension.

We had to squeeze our noise cancellation DNN 30x to fit it inside Chrome.

This was quite challenging and thought it’s a story worth sharing.

You have trial until April 1st. Then it converts to 120mins/week. If you are a student, teacher, hospital worker or a non-profit - reach out to support@krisp.ai and you will receive 6 months free.

For the next 6 months, Krisp is free for all students, teachers, hospital and government workers.

Also, it just went Freemium, comes with 120min/week free noise cancellation.

More here: https://krisp.ai/blog/covid19-response/

(I'm the CEO & Co-Founder)

We have built a DNN which does 2 things on real-time audio:

a) remove background noise b) remove room echo

We’ve then embedded it in a virtual microphone which can be integrated into ZoomRooms or similar products.

Demo with video here: https://www.youtube.com/watch?v=AnoWG1JBe8A

CallerID is not supported yet but it will come soon.

Yes, and hopefully very soon.

Apparently getting access to the microphone stream during calls, even from your own app, is really tough. There is only one provider that we know that implements the concept of "advanced audio filters" and that's Twilio's Voice SDK for iOS.

Noise cancellation on iPhones (and other phones) is outdated as it doesn’t support the form factors people increasingly use. It only works when we hold the phone close to our mouth.

We have built Krisp iOS app to fix this. Krisp is powered by our infamous Deep-Learning based noise-canceling technology.

The app is very simple, yet quite practical. You can join Zoom, Meet, Webex, etc meetings with a single touch and also call any phone number you want, without worrying about background noise.

Intercom button on the right.

Krisp removes screaming child noise quite well.

This was a sensitive use case for the team since I (CEO/krisp) had a baby 3 days ago and really needed Krisp to do my calls :)

my recent blog post had some details about the tech behind krisp:

https://devblogs.nvidia.com/nvidia-real-time-noise-suppressi...

Such feedback from users is priceless. Thanks for it. Krisp is still in beta and we keep experimenting with different DNN models. For example we have a DNN model which maximally preserves multiple voices while removing noise. It's not shipped yet.

Re #2, one thing we noticed is that Conferencing apps themselves will distort the voice when multiple voices are overlapping. Especially when there is also noise. There is not much Krisp can do here since the stream it receives is already distorted. Unfortunately for krisp speaker we don't control the audio stream. Imagine how many times the stream gets signal processed before krisp speaker gets it (noise cancellation in the headset, noise cancellation in RTC, codec, etc).

Re Krisp microphone, the DNN model used here is more effective since what Krisp receives is "less processed/distorted stream".

Please stay tuned, our release cycle is around 2 weeks. More quality and UX features are under way.

This looks intriguing. Do you know if it receives the audio and re-routes or simply changes the routing configuration?

Around 15ms to the end to end call.

Not really. The tech used in noise-cancelling headphones is called ANC (active noise cancellation). It eliminates the noise coming to your ear from your surrounding environment.

In contrast, Krisp eliminates the noise going from your environment to the call participants and vice versa.

It's currently free as it's in Public Beta. Krisp will have a freemium business model after Beta.

It's fixed now. Thanks for pointing to it.

Mobile support is harder since Krisp won't be able to register as a virtual mic/speaker on Android or iOS. At least we haven't found any way around it.

2Hz CEO here (we built Krisp).

Windows support will come in Dec. We are working hard on it.

Windows will come soon. Linux - no plans yet. Yes, it cleans the incoming as well.

This technology is already integrated into Krisp app (https://krisp.ai) and it runs all locally on device.

Disclosure: I'm the author of the blog post.

Not really. What works inside noise canceling headphones is a very different technology, called Active Noise Cancellation (ANC). You don't necessarily need Machine Learning to solve the problem.

The technology described in the blog post is for suppressing the noise which goes from your surrounding environment to the other participants of the call (and vice versa).

Disclosure: I'm the author of the blog post and co-founder at 2Hz.

This is a guest post on NVIDIA Developer Blog. The author of the technology is a startup called 2Hz (2hz.ai). Our passion is to improve voice audio quality in audio/video calls. It's a tough problem but also fun to work on.

Agree, breathing, reverb, noise are all problems and should be fixed. We started with noise and already shipped a product you can try on your Mac. The app is called Krisp (krisp.ai).

Reverb, breathing, voice cutting will come next.

The most natural place for PLC is running inside a codec. So API probably wouldn't make sense. We do have plans to integrate it into Opus/WebRTC codebase at some point.