Of course not, but phrasing it in this way (that phones are a separate exception) minimizes the legitimate critiques of always-listening cellphone privacy.
I'm willing to make some sacrifice so that I can get GPS/maps anywhere I go, ability to make emergency phone calls, etc.
Smart speakers really don't have the same value add.