How do you stop your phone from always listening to you? It has a microphone and an internet connection. The hardware and software are closed source and out of your control. It is also orders of magnitude more powerful than a smart home device and is within listening distance of you pretty much 24x7.

If you are scared about an Echo in your house then you should be keeping your phone, tablet, laptop and 50 other similar devices locked up in Faraday cages at all times.