How is this different than picking up the phone and having your convo go through ATT/Verizon networks? or using your ISP? Both parties can "legally" work with authorities to wiretap you?
Are you worried that the (training) algorithms that run on your voice somehow end up leaking your identity? Or are you worried that someone at Google knows your voice?
Also, not sure if Google has this fact in their ToS but if they do, what is the issue?