It's not your privacy that's the issue (Well, it is, as a user of the device) - it's also the privacy of the persons speaking with you. Single-party consent for digital recording of audio is a controversial topic.
But it's also the privacy of everyone close enough to the microphone that single processing can extract what they are saying. That's 0 party consent that I'm not ok with for always on devices.
Now this piqued my interest, so I took a look at Wikipedia[1]. Interestingly enough, England seems to allow recording phone calls without notifying the other party as long as it is only for personal use and not made available to a third party, but US Federal law does not seem to have such a stipulation, thus presumably allowing people in one-party consent states to post recordings of phone calls on social media without the other party's consent (unless the state forbids that. It's a bit hard to find concrete information on this), which is pretty troublesome I think. However that is only for telephone recordings and not for private conversations, so maybe the laws are a bit more sensible in that case.