Having the microphone always-on drains a lot of power. You would notice it very quickly were your cell phone to transform into an always-on surveillance bug.

You're technically correct that cell phones leak all sorts of meta data that can be used for surveillance, but they aren't always listening. Voice assistants, however, are designed to be always-on and that is the key difference.