Obviously there can be defects, but Google is not getting 100% of the audio from your home.
You are using 'Google' in a vague way which may be why you are confused about what my comment meant. The Google device gets 100% of audio it hears, because it can't magically hear a 'wake word' without listening to all audio. It only sends audio to a Google server after the wake word.
(This is at least true for the iPhone, Hey Siri is recognized locally before anything is sent off to the net)
Re reading your comment, I guess you mean: the sound processing goes through the local microphone no matter what, even if the audio is only forwarded on if they lead with certain keywords, right? In other words, just like how every other voice activated system has to work. The privacy implications are that the equipment could be compromised to actually put the recording in longer term storage, like a bug or something.