You can completely disable Siri or whatever your device's equivalent is. The entire purpose of Alexa is to be listening all the time. They're not equivalent.
I don't see a huge difference between a speaker that's designed to have an always-on mic listening for a code-word and we're just going to trust that it doesn't log anything unless it hears that codeword, and a phone that's designed to have the same, but has a software option to turn it off.
Either way, we're trusting that the device isn't abusing the fact that it has a mic.
More importantly, the difference is in intent. I don't intend to carry around a device that's always listening to me. Anyone with an Alexa device is intentionally inviting Amazon to listen to every word spoken in (part of) their home.
I carry around and iPhone where I've done my best to completely disable Siri so I expect it not to be listening. That's antithetical to the use-case of Alexa and similar devices.
For now. Or so we're told.
Unless a court order (or bribed employee) decides otherwise. ;)