I don't see a huge difference between a speaker that's designed to have an always-on mic listening for a code-word and we're just going to trust that it doesn't log anything unless it hears that codeword, and a phone that's designed to have the same, but has a software option to turn it off.
Either way, we're trusting that the device isn't abusing the fact that it has a mic.