A lot of phones already listen to audio nonstop. Take the Android phones that listen to "OK Google" or the Apple phones that respond to "Hey Siri". Those features require the phone to listen to and process audio nonstop. Sure, if you don't like that then you can turn it off, but I don't understand why people have concerns about stationary always-listening devices moreso than phones with the same functionality. These base stations or hubs are the same concept, except they're present in a room instead of part of your phone.
All other things equal, I think mobile phones are a much greater threat vector than home hubs. Mobile phones are juicy targets with large, complex attack surface areas. See the various remote code execution exploits in Android that could be triggered by sending someone a message. See the various high levels of privilege that applications have that can be used to spy on you. Did you see the article about how Uber had secret privileges that allowed the app to record iPhone screens? [1]
There are multiple reasons why an attacker would want to compromise your phone (e.g. to steal your data, not just record your voice). Many people have phones, so sophisticated adversaries are more likely to invest the energy into developing attacks. There's a lot more to compromise (apps). Home hubs are less likely to see that same level of scrutiny, and they're less likely to be vulnerable to the same types of attacks and issues because they're much simpler devices (they don't run "apps", they don't communicate on the network except with their vendor).
If you're worried about being spied on, then you should also be worried about other people's phones, which you may not even realize are in the room with you! By comparison it tends to be pretty obvious / known to everyone when there's a home hub present in a room. Your phone probably doesn't have a hardware-level light that tells you when it's recording you, like your home hub does. The list goes on.
[1] http://bgr.com/2017/10/05/uber-app-privacy-issues-iphone-scr...