Math professor here.
When I go to research lectures, I sometimes hear that in response to audience questions, although not especially consistently. Some speakers do this more than others, I don't think anyone does it all the time.
When I go to research lectures, I sometimes hear that in response to audience questions, although not especially consistently. Some speakers do this more than others, I don't think anyone does it all the time.
The next question (which is a great one, from what I understand) is: Why do LLMs use these phrases so much if humans rarely use them in written form? Maybe a fair portion of training data comes from lecture transcripts, where such responses are common when responding to direct questions? And/or system prompts are just instructed to be like that?
As far as I understand, it's due to RLHF. The reviewers the AI companies use don't necessarily know what kind of question is a good one, so when the LLM answers "That's a good question!", they tend to rate the answer higher because they like being flattered. Proxy models that are themselves trained on RLHF inherit this pattern. Similar effects contribute to sycophancy.[1]