I get what you're saying, but you're ignoring intent here. They're, literally, using the wrong word, without meaning to. In their language they have multiple "yes" that mean very different things, but they incorrectly use our single "yes" for all of them which, as you're very correct to point out, has a very specific and STRONG meaning. This is a conceptual mapping mistake, not an intent.
They're trying, and slightly failing, to speak a language they took time to learn, but is still unfamiliar to them, my dude. The alternative is that you/I should learn mandarin. I applaud their efforts that allow me to be lazy, even if it means I have to understand some shortfalls in the communication.
If you learn a language, but accidentally use the wrong word in conversation, because maybe nobody has corrected you before, does that make you a liar? Of course not. That's what's going on here.