We have some google nest speakers and wow, they get it wrong so often despite being stationary. They can connect to one person, not another, drop connections, vanish for days until I care to reset them and get them back online. Sometimes the voice changes, haha. Airpods though, I can share them with my family and they work great between each of our devices.
It’s one of those things where I consider myself lucky. As a software developer I don’t subscribe to the “no adults in the room” attitude, but I know this stuff is hard. I’d expect more people to have problems.
The switching itself technically works. The problem is it needs to be smarter to not switch away when audio is currently in use - especially if the mic is in use.
One issue I've seen a few times is if I have something playing on my mac, and a video going on my iPad, they've gotten into a switch loop.