You're still relying on a piece of software to make the wake-word assessment and hand off the audio to the cloud. Now you're just adding more hardware parts to fail.
If your argument is that you trust your software more than Amazon's, then you shouldn't need anything more than a single microphone anyways because why would you surveil yourself?
For the same reason nginx usually runs as a separate user: people make mistakes and security vulnerabilities happen. Security in depth is a good thing.
Like hey, let's install some under developed AI from some unknown company with unknown security policies on top of a device with access to hordes of personal data and the ability to make transactions online.
No hacker will EVER think to use it as an attack vector /s