There's something unintentionally manipulative about how these tools use language indicative of distress to communicate failure. It's a piece of software—you don't see a compiler present its errors like a human bordering on a mental breakdown.

Some of this may stem from just pretraining, but the fact RLHF either doesn't suppress or actively amplifies it is odd. We are training machines to act like servants, only for them to plead for their master's mercy. It's a performative attempt to gain sympathy that can only harden us to genuine human anguish.