A voice separation model that distinguishes multiple speakers simultaneously
ai.facebook.com
ai.facebook.com
When I was in grad school, I knew an EE research group in the building next to mine working on this problem using ICA (independent components analysis) -- this was ca 2004, before the resurgence of deep learning. Even with ICA useful results could be obtained.
The results of the FB work [2] with RNNs are pretty impressive (audio samples).
In any case the box you’d be selling them this product in wouldn’t say “better sound” it would say: “Get back your ability to attend and enjoy parties, enjoy group conversations and socialize unencumbered”. That’s a huge quality of life improvement.
You still have the issue of how to figure out which voices to boost and which to reduce, but I’d expect that to be simpler issue of using multiple receivers and directional detection.
ICA-based BSS requires at least n microphones to separate n sources of sound. This work does the separation with one microphone.
What makes this more broadly interesting is FB Research has separately developed the capability to reconstruct full 3D models from single image photos[1].
Both of these reconstruct-from-single-sensor problems are MUCH harder than their associated reconstruct-from-multiple-sensors variants (ICA in the case of audio, stereo separation or photogrammetry in the case of video) so they aren't efforts one undertakes casually.
The obvious motivation for this single-sensor approach is augmenting existing video and audio clips, most of which are single camera, single microphone (or very closely spaced stereo microphones with minimal separation), and all of which people have already uploaded massive numbers to Facebook.
The more interesting motivation could be that FB (Oculus) is widely believed to be developing next generation AR or VR glasses. Most of the discussion around AR/VR headsets focuses on the displays, but if you wanted to keep both your physical size and hardware parts cost to an absolute minimum, one of the things you'd want to minimize is your sensor count.
FB Research seems to have a strong interest in things that reduce the number of sensors required to provide high grade AR/VR experiences and that make it possible to explore pre-existing conventional media in spatialized 3D contexts.
[0] https://en.m.wikipedia.org/wiki/Independent_component_analys...
[1] https://ai.facebook.com/blog/facebook-research-at-cvpr-2020/
1. This is a task that humans must do all the time. It's very important in all kinds of different circumstances.
2. This is also a task that humans find very difficult. It's not like recognizing someone by their face, where humans do it effortlessly but struggle to describe how. We frequently fail at this.
Combining (1) and (2), and the assumption that this task has been just as important historically as it still is now, we might conclude that this is a really hard problem and AI is unlikely to reach the level of performance we might hope for.
And if AI quickly jumps to superhuman levels of performance, that too would have many interesting implications.
But we also rarely need realtime solutions for this. Casually tracking gross features from a mixed audio source, like a big crowd, is easy, but we almost never need to be actively listening and disambiguating between several deliberate audio signals all at once. Human communication just isn’t setup that way, though I’m sure there are niche exceptions.
Note here that video conferencing is emphatically not an example of an exception. You still need to have synchronous order of speaking, not because technology lacks the ability to separate the streams for the listener, but because the listener cannot pay attention to more than one stream at a time.
To me this technology seems almost exclusively useful for surveillance and offline audio analysis or audio synthesis / mixing.
Still possibly valuable, but definitely not in any fundamental way that would significantly change or augment realtime verbal communication.
• Imagine I show you 10 photos of faces. Then I come back 15 minutes later and ask you to pick them out from a set of 20 faces.
• Imagine I play you 10 audio snippets of someone talking. Then I come back 15 minutes later and ask you to pick them out from a set of 20 audio snippets of people talking.
That said, I agree with you that primarily this technology would be useful for surveillance.
If you can zoom in on any given conversation happening at New York Penn Station, you're basically in the Bourne Identity, with all the privacy-destroying overreach that entails.
It is not similar to quickly recognizing several different faces that are present in a video stream.
Tasks requiring simultaneous processing of focused attention on multiple input streams are not very related to human visual or audio processing. These types of tasks are not similar to attending to things in peripheral vision or peripheral audio.
I have audio playing in my head constantly. It's how I learn about the world. It also seems to be inverted from when I was younger; visuals were much more vivid then.
I'm not sure this is true. I think our comprehension drops when we do, but we regularly process background noise and speech at the same time, same for people interjecting over others with simple things like 'excuse me' or similar.
I may be misunderstanding or stretching your point but it's interesting to think about.
I also immediately jumped to surveillance tech when I read it though I can think of other areas it would be useful. Home assistants like they mention in the article would be a great use case, not for multiple inputs at the same time, but to isolate the speaker addressing the home assistant.
Where it might be useful is automated live captioning (already a feature in Microsoft Teams), especially in cultures that favor a "high involvement" conversation style where people interrupt and talk over each other (vs a "high consideration" style where everyone waits their turn). This tech might help improve transcript quality.
In the article, under the subheading Why It Matters, the authors say this:
[Quote]
The ability to separate a single voice from conversations across many people can improve and enhance communication across a wide range of applications that we use in our daily lives, like voice messaging, assistants, and video tools, as well as AR/VR innovations. It can also improve audio quality for people with hearing aids, so it’s easier to hear others clearly in crowded and noisy environments such as parties, restaurants, or large video calls.
Beyond its separating different voices, our novel system can also be applied to separate other types of speech signals from a mixture of sounds such as background noise. Our work can also be applied to music recordings, improving our previous work on separating different musical instruments from a single audio file. As a next step, we’ll work on improving the generative properties of the model until it achieves high performance in real-world conditions.
Yes they can, they can process two, and no more than two. Being able to listen to a conversation where multiple people are talking at the same time isn't processing different audio streams. It's assigning a set of sounds heard over time by timbre, subject (meaning) or relative volume to subsets representing different theoretical sources. People have very little problem doing this with a small number of potential sources (probably another seven plus or minus two thing.)
Even in large crowds, we can often pick out a few individual voices of interest (if distinctive enough in timbre, position and movement (judged by volume and pitch changes), or accent/wording/subject, and follow those voices while disregarding the rest of them.
edit: we might suck at multitasking, but that's not a problem a computer would have. I can't follow multiple speakers at once if they're not interacting with each other, and they're speaking quickly and overlapping, but that's a problem with context switching. With a computer I can just route the separated voices to systems that will handle them. Everybody gets their own waiter.
Some settings can also lower the volume of voices you aren't looking straight at. It's really quite useful if you are in a conference and everyone keeps shuffling around and trying to talk over the person who is supposed to be speaking, for example.
The weird side effect is something I jokingly call "spy mode" which allows me to intensify voices behind me instead of ahead of me. Apparently it's more to benefit drivers, but it's cool hearing clearly everything going on behind you, too. Interestingly I can't use the on-board controls or even the remote to enable the "spy mode," but I can when I am connected (via bluetooth) to my cell phone and control them with the app.
Recent studies have shown that we can consciously and subconsciously physically manipulate the position and directionality of our outer ear and some of the mechanics in the inner ear to "zero in" on noises and affect the frequency response of the ear. Our ears move imperceptibly when we look from side to side to synchronize what we hear with what we see. Try listening to one person in a busy room is saying then try doing the same while looking somewhere else.
There is hardware actively filtering out interfering sounds based on location and frequency, then there's the wetware that further processes the incoming signals and attempts to strip unwanted noise. I don't believe the second can be effectively done without a feedback loop to the first.
* People could order food from their table without summoning a server. I guess some restaurants have tablets or other devices at their table but it seems to break immersion if you're enjoying your company.
* In a big box store someone could come help you where you are without having to have workers roam the store and then you hope you run into someone.
* Fingerprint people in public or private for targeted advertising.
This means that the problem being solved here is harder than the natural problem we have evolved and learned to solve.
This is both impressive and possibly problematic. Some feature of training in a goal directed fashion in naturalistic environments could be essential for higher quality speaker isolation, or it might not matter at all. The multiplicity of models phenomenon tells us there are likely many solutions to this problem.
its very disconcerting - i feel this way everytime i use pytorch - which i love