If you trust that your phone isn't listening when it's not supposed to, why don't you trust the home assistant (even if it's made by the same manufacturer)?
Do home assistants claim they are not listening when they are not supposed to?
Even if you create tasks that can function through a button press or something they still rely on the cloud.
Voice control locally is really hard though. Or at least it used to be. I thick OpenAI Whisper is available for local/offline usage if you build your own wrapper around it? I tried adding (local open-source) voice to my home system about 8 years ago, and again 3 or 4 years ago and it was very rough. It might finally be feasible with the current state of local AI.
For an example, my home robot was named "Marvin" (from H2G2) and the local detection was so poor, this is my list of "matching" wake words.
['Marvin', 'arvin', 'artin', 'marlin', 'marten', 'margaret', 'why are']
I didn't really value voice control until I had young kids.