I have one in my house in the living room. It basically exists to have an easy way of turning on Spotify. We don't have sensitive conversations in the living room. If someone were listening, they'd mostly get me scolding my children and asking what's for dinner. The might also steal a token to connect to Spotify. My AWS account isn't linked to the same account as my Alexa, and requires TFA, so that's safe.
I wouldn't put this device in my bedroom. I also was less interested when my kids were young enough that I might actually have a sensitive conversation anywhere. I'd considered putting one in my tv room to control the tv, but that's about it.
I don't regard the Alexa as a greater vulnerability to my house than my phone, and I already accept owning a smart phone. I am concerned about the same things you are, but I view it as more of a trade-off than a simple "just don't do it!" attitude.