It's not like there's a law of physics requiring that always be true, you're trusting that to:
A) Be true now, and B) Keep being true in the future.
And it's the sort of claim that is very very easy to lie about.
State adversaries are a different story. I don't doubt the NSA might somehow break into Snowden's Echo, but most people don't have such adversaries.
(I mean, what the hell - deep in the options of Google Now I can actually allow the phone to be unlocked by my voice - but if and only if it is plugged to a charger. They had to explicitly code in a special condition to make Google Now less useful.)
(and yes, I know you meant other kind of "always listening")
State adversaries have repeatedly demonstrated a desire to "collect it all" and sort it out later.
https://en.wikipedia.org/wiki/Room_641A
The entire emphasis of what Snowden revealed was precisely that the method employed by the NSA were not targeted.