That's not the problem; it doesn't matter if your particular device is listening at any particular time. Using an always-on microphone normalizes the expectation that audio in the home might be sent to a 3rd party's remote server. This is important because Kyllo v United States created[1] a bright line test for when a technology is a "search".
Normalizing audio eavesdropping technology in previously private areas will eventually mean use of that technology is not a search, and thus the police/etc can use their own device with similar technology without a warrant to see "details of a private home that would previously have been unknowable without physical intrusion"[2].
For example, the CIA developed and deployed a program to use the built-in microphone on Samsung "smart" TVs to spy on targets, even when it looks like it's off: https://www.theverge.com/2017/3/7/14841556/wikileaks-cia-hac...
It's already happened (accidentally, in this case): http://money.cnn.com/2017/10/11/technology/google-home-mini-...
And it will continue to happen, because "prevent this from happening at all cost" isn't a development mandate for the device, and won't be, because they would prefer to capture everything. Google famously captured wifi network and street view data after they were specifically instructed not to by the court -- why would they fail to avail themselves of information that's right there in this case?