Google Meet rolls out multi-device adaptive audio merging
workspaceupdates.googleblog.com
workspaceupdates.googleblog.com
Outputting audio from multiple laptops in the same room is easy. Perfectly syncing it is harder. Implementing echo cancellation across all of that is quite a bit trickier than regular single-device echo cancellation.
But then treating all the laptop microphones as a kind of microphone array, having to deal with sync issues and phase issues and background noise issues... that's hard core.
Kudos to the engineering team on this one. This is actually pretty amazing.
Related, there are devices you can buy to repel teenagers: https://www.today.com/news/controversial-mosquito-sonic-devi...
This would require measuring equipment, since you can also only hear what you PC can output that you can hear. ;)
Stupid devices IMO. When they first came out I downloaded a sample audio file from the manufacturers website to see if I could hear it. I couldn't.... because they encoded it as MP3 and it was completely filtered out by the encoding! Literally an empty file.
But all of them made by "LG" and featuring a single-chip design are either IR or RF.
I visited her recently and wasn't sure if I had finally aged out of my sensitive ears (34 now) or if her batteries needed to be replaced.
FWIW I wonder if people do not experience physical pain from certain sounds, because people seem to be totally fine with sirens but it feels like I'm having a spike pushed into the side of my head.
This no longer seems to be the case, as I'm living right by a major junction and get random full volume sirens at least six times in the average day. I hate them, but they don't hurt.
it seems to occur only late at night but could be wrong... i had assumed they were for driving rodents away but i guess not?
And yet Google can ship it anywhere on the planet via HTTPS and webasm... Things that make you go hmm.
This essentially replaces that expensive proprietary hardware with a matrix of laptops, and essentially every user gets a mic.
But it only works on laptops right, not phones?
But I don't necessarily know that Meet is trying to tackle all this? Are they using the mics as a microphone array & processing signals across phases? Could be missing it but I don't see that they said so. Perhaps they're just picking the loudest mic for a given speaker? Or any of a dozen other simpler tactics?
If the distance from the microphone to an "unwanted source" is three times the distance asthat from the microphone to the source phasing likely wont be an issue.
There's always caveats with engineering but it's a decent rule of thumb assuming equal volume sources... I can imagine it's not too hard to detect that anyway, weve been able to do realtime fft for a very long time.
If they're actually able to "sync" (again a poorly defined term) given the problems associated with network latency and different hardware it would border on magic.
But they can be quite fast.
I wouldn’t be surprised if they were and audio is routed through a neural network before relay.
That's not to diminish the accomplishment or say that it's easy (or that I could do it), but I don't think a neural network is necessary here.
Too many times these kinds of services are wrapped up at the application layer, where really they belong in the operating system. For example, wouldn't this be a perfect thing to implement as a plugin for Pulseaudio, or JACK, or even .. VST?
(Disclaimer: I work on high end microphone and audio products at a well-known hardware manufacturer of such, where much more effort is being made to make the devices, themselves, smarter ..)
I would honestly try to sync audio output based on a shared time reference, something along the lines of what AES67/Ravenna/Dante does but you can be a little more lax and use ntp or system time since you don't need to be sample accurate.
For the microphones that would be a little harder but you should be aware it's not that tough since a few high end manufacturers have phased microphone arrays for videoconferencing. You could probably get close though but the fact is you need the audio from all the sources in a single location for processing and do phase analysis on it and possibly find an optimal delay for each by checking the group delay.
The advantage they have is some latency is acceptable and they don't need to do it on a low power device.
I don't see what this has to do with Gemini but maybe that's just marketing...
The fact that phased array microphones exist has nothing to do with the point we are discussing, which is audio coherence across heterogenous devices whose only real connection is a web browser.
I'm thinking more some sort of system with a sync point registered per device and using that as a time reference.
It's not inconceivable that they could easily detect multiple devices in a room and find a sync point based on microphone input from a speaker.
Once you have a sync point found you can then set a delay on all devices to try to match that sync point. Nobody said this is easy or everyone would be doing it but it's simple enough.
The phased array microphones is more a pipe dream but you wpuld absolutely be able to do something approaching that with multiple devices on a single room depending on how accurately you can predict microphone location within the room. Im reasonably sure you could start by just using the closest mic and then over timr as you improve sync you can try to use multiple.
As I said they get every single audio stream in and out into their servers and they have full control of the audio the tab is playing and the timing of that.
I don't see this being any different to what the likes of Sonos/Google Home/ Apple Home etc are doing with synced appliances for stereo/ multichannel devices, it's likely significantly harder because it's heterogeneous devices as you said.
All that doesn't answer my question of how you would do this at the OS level? You don't have any of the required information per device, only the central server has even the hope of having all the relevant information and control.
It is apparently a good example of something that needs performant neural nets in the cloud to solve. At first glance it looks like a low-level hardware-firmware problem. Market conditions prevent solving it at that level though, so we had to wait for the right combination of resources, new signal processing and heavy cloud compute.
Zoom and Teams always frustrate me, even though they're what I use far more often.
It's too bad that video conferencing is largely a game of who has the most boxes checked - IT admins and A/V folk are the ones that need to be convinced, and Meet just doesn't ...meet... their needs
It's great, there are just some tiny UX niggles that could be solved very easily but haven't been, yet.
It takes so much messing around with to actually get to the call, with meet it is one click every time.
Install the fat client, no thank you. Connect computer to audio? Why is that even a question? Zoom is living in the 2000s.
1. change Meets call layout to one video (presentation) only, disabling participant watching
2. press F11 to make browser go full screen
3. press Ctrl+Minus multiple times to decrease UI elements which makes the shared video near full screen
I also think this is perhaps the top issue i have with Meets. The current UX does not consider the fact people share screen most of the time. having ability to go full screen to get 1 to 1 scale with some essential floating menus. Even more recent years when remote teams are the new normal.
https://chromewebstore.google.com/detail/picture-in-picture-...
Zoom also transmits much higher quality video and screen share feeds than meet does.
This new feature is very impressive although, most echo cancellation is "mute everyone except one speaker" which leads to walkie-talkie style half-duplex talking, which really hurts normal communication flows. I wonder how they do it here.
My daughter’s school uses Zoom for her IEP where at least one attendee is remote that day. Not a fun experience.
One of the interesting things for me from a technological perspective is whether or not we're converging on what the ideal set of features for a "video phone" would be. If there was a broad enough consensus on the core feature set I would hope an 'appliance' version would be available which would eliminate needing to use a general purpose processor (and all the risks that entails) for this sort of meeting.
[1] From your response I infer that you value "no native code" and "sandboxed javascript" highly which guides you to the choice to use Meet, vs someone who might stack "User Experience" more highly than those two and end up at a different choice.
At least personally I refuse to make a Google account bc it requires giving Google my phone number... But I know that's a bit of technoludditism in the current zeitgeist
Perhaps it's meeting specific, or organization specific?
https://support.google.com/meet/answer/9303069?hl=en&co=GENI...
usually there is need to some third party extension/application that have usually does not support Linux
or if Linux is supported, they explicitly ban Wayland/pipewire (I'm looking at you slack)
but for need they stick to web standards therefore this works in any web browser you might have
also it tends to not consume every CPU cycle you have (slack again) so you can do so something while you eg screen sharing
As it is, we're gradually shifting more things to Zoom where that's either part basic functionality, or part easy to automate as things like Zapier are well integrated into the API.
We only use Meet for team meetings now, and we did use Gong.io to solve the above... but Gong is pretty expensive just to allow those who were out that week to catch up on a recording when they're back and if they care to.
I wouldn’t want “autorecord” though - it’s better for culture and rapport to not feel like every conversation is “on the record.”
Autosharing can be easily solved by simple Google Apps Script. I scan my personal calendar and share recording to Slack channel when I detect group meeting with recording available. Mail me, so I can arrange something for you.
Autorecording - yeah, this is missing. There is paid Chrome Extension which do that, but I have never tested it.
Having said that, so many people don't record meeting I or someone else can't attend. This is annoying. It should be automatized so that if you miss part or all of a meeting you can still watch the record as long as you were invited.
My memory is not great. I'm starting to have more calls with clients and it would be great to have a record rather than trying to take notes and conduct a conversation at the same time.
> It should be automatized so that if you miss part or all of a meeting you can still watch the record as long as you were invited
People generally don't like knowing that everything they say is being recorded
If there is anything I want to keep, I will usually take notes but sometimes I cut the small part I need and store it in a special folder. I just wipe once in a while the main "obs" directory, every other week or so. if I haven't felt the need to play back a video, I doubt it will in the future so I don't keep a lot of retention, it is pretty much only to help me when I am getting distracted or multitasking during a meeting. Most of the time when I feel the need to play back the video, it is immediately after the meeting because I know something important had been said but wasn't 100% focused and want to be sure I haven't missed anything.
But I don't have lots of meeting, a handful a week usually. I don't need special hardware, obs seems to be heavily multithreaded. It might hurt the battery usage if I am not plugged but no core is going very high in term of cpu and I don't feel any slowness. I am recording in the 2500/160kbps veryfast(medium CPU usage, standard quality) setting at 1080p, it takes like 1MB every 3 seconds.
The process is exactly like that of streaming a computer game along with local microphone audio, except one pushes the "Start Recording" button instead of the "Start Streaming" button inside of OBS. There's got to be a million (or more) howtos written on the subject.
Hardware-wise, it's pretty straight forward: GPUs (including the ones that are a part of most non-Xeon Intel CPUs) have been up to the task of realtime video compression for around a decade or maybe more, which allows for the heavy lifting to be done in specialized silicon.
A bigger concern than the technical practicality might be legal concerns that generally surround audio recording.
For instance: In my state, I am permitted to record any conversation that I am a participant in -- I don't need permission from anyone but myself, and I don't need to notify anyone.
But in the US alone, there's also 49 other states worth of laws on the subject, and they can vary quite a lot.
Getting logged in is always a problem. Setting my name permanently is literally always a problem (I have zero idea why). Viewing someone elses screen always stinks because the scaling is just not good.
On top of that you can't compare Teams to Google Meet because literally every organization has theirs configured differently. For some organizations, I can use my phone for audio. For others I can only use my laptop. So it's never an Apples to Apples comparison. Sometimes you have access to chat history; other times if you reconnect you lose chat history. The differences inside a product go on and on.
Not quite sure about this. Gmail (both personal and company accounts) is IMO a great email client with loads of handy features, and I always felt Google Docs/Sheets/Slides/Drawings were well put together. Are they the best in class? I guess it depends what you are looking for. Could Drive be faster? Yes, that would be nice. But "amateur hour" sounds like we are simply using different products.
HN: https://news.ycombinator.com/item?id=30227588
Article: https://medium.com/@laurajavier/google-slides-is-actually-hi...
Regarding support, that is not an issue I have never needed it for Google, but occasionally have needed for MS Office. So I can see why if you are using Office you would value that.
Note sure if that’s because I’m not in the US.
In my experience Zoom is always slightly better, but at a nitpicking level. I use meet day in day out, and fallback to whatever we have in hand if it's unworkable (pixelation would hit that line), and never saw zoom or Skype or discord being significantly better at these times.
But Google Meet is pretty good otherwise. Zoom is good too. All others I have used have been quite annoying to use.
Unfortunately, it's only a matter of time until one their VPs decides to "improve it" with the usual imbecile ideas they come up with.
>Google Meet now scans all your files so they can get shared automatically with all members on the meeting when it is appropriate to enrich the conversation. You cannot opt-out this feature.
Such a nightmare. Quality of meetings went downhill and employees were penalised for secretly using Meet for ad hoc meetings.
Why would someone ever be secretly punished?
This is very common anywhere that handles sensitive info.
Also I swear the audio latency is worse with Zoom -- I find myself accidentally interrupting people more often.
Team's load time is atrocious, though, by comparison.
I worked for a company that was headquartered in London and had satellite offices in Spain and Germany. After we all went remote during the pandemic the EU offices said they felt so much more engaged with the rest of the company because they were no longer disadvantaged by default for not being in HQ and in person bad habits were penalizing them
Its definitely doomed to fail if CEOs want you to think it's doomed.
Your comment is handwavy with words like "feel" and "bad habits". There are real issues with remote work, but there are also ways to mitigate the downsides of it. It's easier for some people to just dismiss the idea entirely and pretend that in-office work is the better alternative without any problems. I have definitely seen that happen in my company and here.
I have worked there fully remote jobs and the ones that did it best fully leaned into being remote.
There is no hard or fast rule, I've been in a team social where I was the only person remote and it still felt relatively natural. I've done a fully remote follow a recipe and cook at home session which was also pretty fun.
I've also been in places that do the bare minimum of what constitutes remote "oh we use Zoom and screenshare" and dictate to people where everyone is cam off. And the difference is night and day.
I think that a little bit of cargo culting wrt remote etiquette is probably a net good thing because I posit many people still don't know what good looks like.
Full single device for everyone would mean either everyone loudly shouting at their desks from within their noise canceling headphones or the company giving everyone an office with a door. As even Google doesn't do this for their employees, this is the next best thing we can do to save the sanity of our coworkers and respect remote colleagues dialing in by including them in a shared room meeting.
It's a great idea, IMO. You can fit 10-12 of these in the same space as a regular conference room. This solves exactly the problem you're describing.
So it's kinda like sitting in a conference room, except you're talking to other people via a shitty audio quality with latency, random noise, talking over each other due to not seeing each other. You don't even have the benefit of a more ergonomic personal setup like more monitors. When people are in the office, let them do their meetings in person ...
The meeting room (should) have a much higher quality mic than an attendee's laptop. You want that mic.
Worse, when attendees use their mic, someone in the room asks a question, which remote viewers cannot/do not hear, because it is not picked up by the mic on the laptop. Or they join and don't mute, resulting in feedback loops.
I do feel the need for Google's feature though: the pandemic has meant a reduction in offices, so now we're ending up in "co-working" spaces, which — despite this being the bread and butter of the business — have in my experience alarmingly poor quality meeting rooms. To the extent that those present have given up on them. Literally, we were in one with no cabling. We asked the company for a cable, and they gave us a DP cable, but the room was HDMI. In these situations, yeah, nobody is going to want to waste the time trying to deal with getting the coworking space provider to deliver a quality product, and features like the one here help make up the difference.
As far as Teams vs. Meet, it seems like Teams is great when you are working with people that have climbed it's learning curve. Teams is also filled with UX paths where it takes one or two extra clicks (and thoughts) to do simple things like share a file.
Can you give one example?
1) You can call someone in Teams. Sure, Google Chat has "Start Meeting now" but it's very passive. I know some will see this as negative but ringing has massive advantages around UX.
2) Chat, Teams will persist chat after meeting ends. In fact, depending on how you hold the meeting, it might dump the contents into chat channel so it's preserved in more open manner.
3) You can have visible meetings. You can start a meeting in a Teams chat channel so everyone can see the meeting is going on. It creates that in the office feeling of two people working on a whiteboard nearby that if topic interests you, you can join in.
4) Sharing Documents, Add Word Document to meeting, everyone is granted permissions. Done.
The fact so many Google Workspaces companies have Slack is just frustrating. You have these two products that barely talk.
True, but Teams is so buggy and had in many areas has no UX per se.
I'd take for 2 great products (Slack + Meet) with bad integration vs great integration of shitty products.
They do not integrate at all on these topics. Slack really could expand here (outside of huddles).
Auto-detect people not using headphones, and prevent them from speaking. Until they put some one.
Ideally showing them some customisable scolding message.
So far, any feedback / echo cancellation I've encountered just makes everybody's life miserable. Degradation to non-duplex voice (because all except the loudest speaker are attenuated down), "seaside noise" effect", etc.
This is the reason why a phone call over GSM or landline still often "feels" better than any HD video call with people on screens: Low-latency duplex audio.
Most* of this goes away if you just wear headphones.
Maybe this can be fixed by making the algorithm way more complicated, as the announced feature does. But I'd be surprised.
[*]: "Most": 2 people with headphones sitting near each other still cause echoes for each other and other participants. Fixing that is truly novel, and needed even for headphone users.
I was shocked to learn that the latency from laptop to ear is higher than to the other side of the planet.
Most people don't notice the adverse psychological aspects of this:
People get really annoyed by when constantly talking over each other. It subconciously makes you dread talking to the other person.
When everybody is using wired headphones, you suddenly like your coworkers more!
We should probably get off hacker news and just start building a teleconferencing app.
This is the part where I mention how Bose, despite making a $300 headset, didn't put a pin/wire on the wired connection for the mic.
(I usually use the laptop mic nonetheless, though, because for some reason if you want to record audio with bluetooth, the quality of the audio output becomes potato.)
I love my WFH setup with a dynamic right in front of my mouth and have the system properly rung out so I can monitor myself with no feedback. The cartoid pattern greatly reduces the sound coming from the off-axis speakers and the noise gate takes care of the rest. Obviously this isn't for everyone though and good luck having something this in the office unless you have a room with a door.
> Available for Google Workspace customers with the Gemini Enterprise, Gemini Business, Gemini Education, Gemini Education Premium, and the AI Meetings and Messaging add-on.
I have yet to decide whether this is more ridiculous or sad.
also, in a previous iteration of google's AI branding, meet had a feature that would create llm-generated meeting notes for you. i'm unsure if this still exists
https://www.theverge.com/2023/8/29/23849056/google-meet-ai-d...
> and the AI Meetings and Messaging add-on
This might actually be using a decent amount of data center processing to handle the merged audio performantly.
If that's the case, it could make sense that it belongs to a higher tier.
Now fixing search, that would be truly mind blowing.
What you're running into is almost certainly noise cancelling. It's turning off your incoming audio so it doesn't intrude on your microphone picking up your voice.
Disabling or tweaking noise cancelling or audio modes (i.e. to "headset") on basically every meeting software will eventually get you a combination that doesn't do this. It's sometimes a bit hidden though, and many have chosen to just say "everyone gets maximum noise cancelling" rather than trying to guess based on your audio device(s) so it doesn't always do it automatically or obviously.
I was talking to someone who specialises in corporate communication at a conference last week and she confirmed my feeling above as well. Her thing is “talking over each other is a key part of human interaction and forcing one at a time is stifling and unnatural”
You could reduce sample size, and send fewer samples per packet to reduce total delay, but overhead goes way up (overhead is near 50% at 20ms samples, one per packet). In theory, you should be able to do something nice for people doing audio and video by including audio on the video packets, but it's not simple, so I think most conferences don't do it.
Not triplex or quadruplex etc.
In other words, if you're having a 1-1 video call, both of you have your audio working at all times.
But as you go to 3, 4, 5, 10, 20 participants, it generally continues to be just max 2 simultaneous audio streams, determined by whoever has been the loudest recently.
Unless you turn on special features like music mode etc.
This is a feature, not a bug, because otherwise background noise and sounds would start adding up to become intolerable. (Why we usually try to intentionally stay on mute anyways, so we don't accidentally become even just that second audio stream.)
Better to use Meet than Zoom if the Chinese are in your threat profile or a larger threat.
super cool tech, but was hard to convince my org to use it over Zoom/Meet
The functionality here seems interesting, but I assume only works well if everyone or almost everyone brings a laptop. It probably won't work well for those situations where only one or two people take their laptops to an in-person meeting.
In products like Meet, JavaScript just acts as a glue language that sticks all these APIs together. All the heavy lifting is actually done by the browser, and modern browsers are some of the most efficient language runtimes humanity has ever built.
I'm disapponted when people send me Zoom meetings or want to meet on Slack. Google Meet is a 10x better experience.
I know that it's possible to use Zoom and Teams anonymously. But for Google Meet, I assume providing a (verified) phone number is required.
Or mid-prod outage trying to get some engineer to type a command to run for you, and they can't dictate from speech to terminal, because they don't know shell/Unix commands at all. Then I can just write the command on their screen, for them to copy character by character… (to some extent, chat solves this. To some extent. I also wish we could get some basic markdown.)
Though I guess in a hybrid meeting they'd make it harder to hear the people in the room. (But you'd hear them through the laptop, normally? I guess there would be an uncanny delay...)
Cool, so now I can get high fidelity audio of the guy in the corner surreptitiously trying to eat a bag of chips on Google Meet now too?
So basically unaffordable for most organisations.
What does this mean?
We use Meet a lot, and I'd never noticed that happening before and wondered if it was due to some setting.
In Discord, you can actually view your audio in reference to the threshold, which helps get that setting set appropriately for the hardware you have. No such luck for the "real" VC apps though.
I however think (emulated) stereo sound and low latency would do wonders. Sadly this feature here will only introduce latency.
Feedback cancellation for external speakers would also be amazing, so you can get rid of your headphones at home. Close up mic and noise gate works, but is fiddly (and no easy noise gate and compressor on linux...)
I remember using that tool in 2022 or so, multiple people in a room with this kind of audio sync / cancellation.
because that sounds amazing
I hate that I have to always manually tweak it through dev tools to make it usable.
How many 400k TC engineers and misguided product managers does it take to make that change? FFS
It can happen on any service and it means the echo cancellation on their end got confused -- it's an adaptive algorithm but occasionally adapts wrong.
Best way to fix it is to mute yourself for ~5 seconds and then unmute, which should reset their echo cancellation algorithm.
If that doesn't fix it, then have both of you pause and not talk for ~5 seconds. This should absolutely reset it, as the echo cancellation algorithm has now definitively learned what silence is supposed to sound like.
Meet has had echo cancellation from the start.
This is multi-device adaptive audio merging, as the title says. Completely different, and far far more impressive.
I don't trust Google with barely anything these days anymore (except Gmail just because it has been so long, and Maps), but Google Meet is the one thing that I prefer Google's solution over anyone else's.
Meet is just so much better than Zoom, Teams, FaceTime, WhatsApp Video, etc.
I'm so glad they are tackling this specific issue. Pretty amazing feat if it works well.
So much better? Does Meet have instruments for whiteboarding or on-screen annotation?