Real-Time Noise Suppression Using Deep Learning
devblogs.nvidia.com
devblogs.nvidia.com
I think there is a huge market for improving sound quality in video calls.
For me, roughly every second call I make is somehow harmed by some kind of "bad audio" problems. Breathing, reverb, noise, clipping, too silent, there are so many things that can go wrong.
And this really harms the productivity of video calls.
I have started collecting and building tools to detect all of these sources of bad audio and am collecting them at https://www.tinydrop.io Maybe these APIs can help people to improve their setup. But if software like Nvidia's comes along and just fixes the problem once and for all - that's great as well!
This is a guest post on NVIDIA Developer Blog. The author of the technology is a startup called 2Hz (2hz.ai). Our passion is to improve voice audio quality in audio/video calls. It's a tough problem but also fun to work on.
Agree, breathing, reverb, noise are all problems and should be fixed. We started with noise and already shipped a product you can try on your Mac. The app is called Krisp (krisp.ai).
Reverb, breathing, voice cutting will come next.
Something struck me about the sample video. The very first sample included background noise, but it was very easy to understand regardless of the noise, probably because it was recorded by a pro microphone rather than a phone. Every other sample was far more difficult, regardless of noise removal. Noise removal doesn't really seem to help; in fact, any imperfections in the noise removal process actually make the audio more difficult to understand because I have to guess not only the speaker's voice and the noise but also the algorithm for noise removal.
What does help me is low frequency pickup. I think the first sample is easy because there are plenty of low frequency components that are later lost through the phone.
Low frequencies are presumably difficult to pick up due to the size of the microphone in a phone, but could there be a way to restore those frequencies through audio processing? It would be interesting to analyze the response of specific microphones to specific low frequencies and find patterns that an audio processor could use to restore the low frequency components.
Anyway, kudos for doing some very interesting work. I don't know how representative my experience is.
As someone who works with speech content, this seems unusual. Typically, low frequencies are reduced because there's not much useful voice signal there—for example, NPR typically rolls off frequencies below 250 Hz.
Here's something concrete: the first phrase in the video ends with "small demonstration", but starting with the second instance, I distinctly hear "sall" instead of "small". In the version with the noise, the "m" sounds like an aberration of the noise and is detectable. With the noise removed, the "m" is replaced with a blip that sounds like an encoding error.
But here are some toggle options I would want a system like this to do (enabled by default):
* Do not send whispers. If I am a primary speaker, and I switch to address someone local to my side of the call via a whisper, that audio should be effectively muted to the other side.
* Focus muting. If I look away from the screen and begin addressing someone off camera, away from the mic, mute that as well.
* Bark and siren filtering. Specifically able to ID and mute barks and sirens. (Planes, motorcycles and trucks would be awesome)
---
What is your imoressikn of the company and app Temi?
[1] - pactl load-module module-echo-cancel aec_method=webrtc
I was trying to make a program which would FFT sounds from my mic and trigger on certain frequencies or combinations of frequencies. The ideas was to have audio files on my phone to act as a sort of poor mans remote control. Yes I know there are wifi and bluetooth ways to do it but I wanted to experiment a bit with sound.
Anyway I'm pissing around for hours with packages and settings and I can't get the damn thing to work. Turns out my computer came with some super fancy beats audio^tm sound system which actively suppresses a microphone input of constant frequency under the assumption its an unwanted buzzing sound.
All features are someone else's bug.
A mic element costs about thirty cents, and the processing power required for noise cancellation already exists in the CPU of the mobile device.
I think it's particularly interesting that Amazon has made microphe arrays particularly cheap, due to Alexa. MiniDSP offers a microphone array for under $100, which is an unheard of price considering what these cost ten years ago.
https://www.minidsp.com/products/usb-audio-interface/uma-8-m...
A related idea in radar is synthetic-aperture radar (SAR).
A lot of the interesting things in audio were inspired by radar. Dan Wiggins at Sonos used to work on radar, and Don Keele created a loudspeaker technology called "CBT" that's based on radar technology.
Because microphones are basically the inverse of loudspeakers, what works in loudspeaker arrays can also work in microphone arrays.
When you record with a single microphone, you are going to pick up a great deal of background noise. This is because the mic will pic up the person speaking AND the background noise; there's no way to differentiate the two.
With two microphones, we know the following:
1) we know where the microphones are
2) we have a general idea where the persons mouth is, because we know how they hold the phone
Based on that, we have a good idea of how long it should take for the sound to arrive, because the speed of sound is a fixed number.
The first time I ever heard a dual mic phone was when one of my coworkers made a call from the inside of our data center. Typically, he'd have to shout into the phone, because the data center was so noisy, and worst of all, the noise was completely random and broadband. But with dual mics, poof, background noise is gone. It was almost like he was speaking in a quiet room.
Amazon Alexa takes this quite a bit further, and uses something called "beamforming." What beamforming allows you to do is to determine WHERE the person is in the room, based on the arrival times of the sound. It's sort of the inverse of a dual mic setup; in a dual mic setup we can 'clean up' the signal because we know where the person speaking is. In a beamforming arrangement, we can use the arrival times to FIGURE OUT where the person is in the room.
If some security company was clever, they could probably use a beamforming microphone array to train a camera on people in the room.
And keep in mind, Alexa beamforming is two dimensional, but you could go crazy and do a 3D beamforming array if you wanted to! (Alex only knows where you are on a horizontal plane.)
Obviously there's costs to running servers somewhere, but that hasn't stopped companies from making similar decisions for a variety of other services.
What is that on a watch? An inch? Maybe one can be put on the band. Also on a phone the other mic is on the opposite side of the phone so it isn’t directly in the line of fire from your voice.
I downloaded the Mac app they provided [1] which I highly suggest everyone with a mac tests out. I ran it on my old MacBook Air 2013 using daily.co. It worked like a charm. Definitely using this in the next group chat, where there is always someone who forgets to turn off their microphone.
One cool side effect is that it actually removes the reverberations that happen when you have two computers on the same call, where the mics keep picking up on the output of the other computers and a large high-frequency noise happens (that I'm sure we all have experienced). The system simply removed it and I didn't even know it was there until I turned off the app.
Amazing work and I really hope that Skype, Apple, Google etc implement this into their voice apps, or even phone providers build this into phones. Maybe in the future, we actually can have phone conversations in windy weather and on the streets.
[1]: https://krisp.ai/?utm_source=Nvidia%20blog&utm_medium=downlo...
COMMENT FOLLOWS
davitb 18 hours ago
Disclosure: I'm the author of the blog post and co-founder at 2Hz. This is a guest post on NVIDIA Developer Blog. The author of the technology is a startup called 2Hz (2hz.ai). Our passion is to improve voice audio quality in audio/video calls. It's a tough problem but also fun to work on.
Agree, breathing, reverb, noise are all problems and should be fixed. We started with noise and already shipped a product you can try on your Mac. The app is called Krisp (krisp.ai).
Reverb, breathing, voice cutting will come next.
It's really hard to speak when what you say comes echoing back.
[1] https://www.sageaudio.com/blog/pre-mastering-tips/phase-canc...
So, if my voice is distorted in a broken line, it would be reconstructed from the text and reconstruction would sound like me. I guess it will be the ultimate 1kbps codec.
It didn't work out, and they went bankrupt.
Can't help but notice how well Nvidia is positioned for what appears to be a growing wave of demand for GPUs. Surprised this hasn't reflected in their share price (feels like they could be the next Intel, but what do it know).
Not that Nvidia is poorly positioned. In fact, I expect that if dedicated ML chips work out, Nvidia will also put one on the market.
Already done. Tegra Xavier includes DLA (deep learning accelerator).
Not really. What works inside noise canceling headphones is a very different technology, called Active Noise Cancellation (ANC). You don't necessarily need Machine Learning to solve the problem.
The technology described in the blog post is for suppressing the noise which goes from your surrounding environment to the other participants of the call (and vice versa).
Take An ANC with 0 latency, that stops cancelling noise at 8 khz, and add a 50 usec latency to it, now will stop cancelling noise at ~1.5 khz.
But This article talks about 20ms latency.
How did you calculate this?
This sort of noise canceling tries to remove the unwanted noise that is already mixed in with the wanted sound in the same recording. For example, recording a podcast on a noisy bus ride.
The only way this works is for it to be built into each device/OS (ie: firmware shipped by manufacturer). If the tech has merit, we'll be seeing it in a couple of years, whether that involves an acquisition here or independent R&D/patenting.
I can see why the cloud processing makes sense for certain applications / licensers / acquirers (e.g. a VoIP provider like Xoom), but voice comms is really the domain of smartphones, and my hunch is most are plenty powerful enough to do this processing locally.
> devblogs.nvidia.com uses an invalid security certificate. Certificates issued by GeoTrust, RapidSSL, Symantec, Thawte, and VeriSign are no longer considered safe because these certificate authorities failed to follow security practices in the past.
Those are some pretty big names. Names where reasonable companies could believe that nobody would ever dare enforce the rules against them, because it would break the Web.
Who says nobody ever got fired for buying IBM?
Heck. If the for-pay CAs keep screwing up, Let's Encrypt could become the sane, reasonable, conservative choice, even among the most Enterprise of Enterprise Enterprises.