You have a thousand data points mined per day - an Echo or Google Home are no different from any other service that you may be interested in using. Especially as these devices are not "listening in" until their keyword activates them.
You're doing it wrong. ;-) It's easy to leave your tracking device plugged into the wall at home, just like a landline, or put it in "airplane mode" when you don't want to be tracked or bothered.
The only way to be certain that your phone is not tracking you, 24/7, would be to do as you suggest and leave it elsewhere. And even then, it can still be used to monitor conversations in its vicinity.
I'm not overly concerned about these things, but it's good to be aware of the ways that your device can track you.
[1] https://qz.com/1131515/google-collects-android-users-locatio...
[2] https://www.seattletimes.com/nation-world/even-if-theyre-off...
Having a phone isn't a free pass to letting business into the home that will in the end use your habits to sell you those habits at a premium.