For example, just this year my always-on, always-near-me iPhone was subject to an attack where anyone could call me on FaceTime and gain access to my microphone. This attack vector does not exist on my Google Home.
So, why are smart speakers much better at invading privacy?
For a phone the assistant is an optional side-feature and disabling it takes almost nothing away from the utility of the smartphone. You're still left with a smartphone.
Streaming audio is not seen as suspicious.
Also they aren not phones and not using wifi, not LTE/4G uplinks, making it more easy to stream audio without hitting bandwidth caps or using too much battery.
It's way much easier to notice if a phone streams audio often and unnecessarily.