Mine wouldn't? That's an obvious phone scam, they would call me directly before wiring $500 to some unproven weirdo over the phone.
> how the fuck we are support to assume people won't give emotional response?
For the same reason you assume people won't immediately become greedy when a Nigerian Prince emails them about an excellent opportunity to earn a few million dollars. Common sense has to play a part or else you're going to be manipulated with or without AI.
They ask me what my middle name is, and my AI counterpart stops. 10-15 seconds go by, with the halloween chain-rattling CD spinning away in the background. The AI answers correctly in stunted breath, and my grandparents hang up so they can move on with their day. They've been getting calls like this for the past 30 years trying to convince them a family member desperately needs their Social Security number over the landline. If they fall for an AI-generated voice from an unknown caller I'd be genuinely surprised. These are decades-old social engineering scripts, people.