Of course they record what you say, and of course a human is listening to it. How else would they train their models?
I also believe Amazon when they say that it isn't recording unless the blue light is on, and that only comes on when it thinks you said "Alexa".
I mute the device when I'm having a private conversation. If it activates otherwise, I'm pretty aware of that.
I guess it just doesn't really bother me. I have a pretty good idea of what it's recording.