You can disable the voice assistant on a cell phone and many phones come with it disabled by default. It is up to you to have it on.
It's not something that I am OK with. It's a compromise.
I am excited for the librem phone[0], but I can't afford something like that at the moment.