The device does continuously monitor (capture) audio, but the detection is done locally and data is only transmitted to the respective service after the keyword match.
This is reasonable as long as you trust that the device is performing as designed (and that the design doesn't include any undesired behaviors).
The privacy concern is that it'd be trivial for someone to use amazon's firmware keys to sign a new firmware and flash the device remotely that would indeed make it upload continuously. Normally that'd be far fetched, but, with Amazon's connections to intelligence agencies, it's more of a valid threat than it appears on its face.
Such a thing would be trivial to detect with proper network monitoring because an audio stream is pretty simple to trigger an alert on, but, almost nobody is doing that because typically if you're worried about something like that you're not going to buy a device like this.
> it'd be conceivable for someone to use amazon's firmware keys to sign a new firmware
However, I disagree with the word 'trivial' and the framing it implies. It would require a non-trivial amount of effort for Amazon to do this; both to keep it secret within Amazon; and to build, sign, and ship a custom firmware; and to avoid detection by changes in the device's bandwidth patterns.