Infosec conspiracy theories are different from normal conspiracy theories because while normal ones are pretty much always bullshit, infosec paranoia is always proven sensible on a long enough timeline. Yes, there will be an incident sometime soon where law enforcement builds a case through evidence gathered with a virtual assistant using AI or something to sift through all that audio. The constitutionality of this evidence-gathering will be superficially questioned but it'll end up getting admitted anyways, or used sneakily via parallel construction techniques. Assuming it hasn't already happened.
They've already done it with phones. Providers and manufacturers used to resist, but after lots of murky "national security" legislation they can't really refuse anymore. Ever read about how the stingray was first discovered, and that law enforcement was sneakily using it extra-legally to build cases? Now it's pretty common knowledge that your phone can be tricked by a fake cell tower and there's nothing you can really do. But does anyone give a single shit? Nah. It's normalized. Every successive technology that is developed and added to the panopticon toolkit will make us more and more accustomed to it. There's been enough overreaches and abuses of technology already that you shouldn't have this blase attitude about it. But that's not how people really work.