Ask HN: Is AI sycophancy a way to reduce compute rather than make users happier?

2 points·amichail··4 comments

Of course you would like the AI to be more critical about your ideas but to be usefully critical requires more compute, right?

Current AI feels like talking to a human expert who answers quickly without really thinking about the matter much.

View on HN

Regarding being "usefully critical": something I've noticed, and it seems to happen more with cheap or "instant" models, is that it will nitpick seemingly minor things. Often it's things that aren't even all that important to the point of the discussion, but it might fixate on it, and low-key argue about it (but in a circular "I'm not saying X but I'm not agreeing with Y" kind of way).

My theory is that sycophancy is just intended to prevent the AI from spiraling into a loop of ineffectually "arguing" or fixating on unimportant details, because it's both kind of annoying when it happens AND it's obvious that the model is spiraling and burning tokens uselessly when it decides to be uselessly critical.

Feels like both the sycophancy and the nitpicking come from it being a RLHF-ed probabilistic model of text continuations, in which generating a continuation which politely pointing out common minor errors is generally treated as an optimal response, and pointing out that a subtle flaw makes the whole exercise futile isn't necessarily, even if the agent has the ability to identify that flaw and extrapolate its consequences for the whole project which a cheap or instant model likely doesn't.

Maybe but the fact is too much sycophancy makes me run away (especially on Gemini )

yes if you want it to be usefully critical, its gonna cost you. its gotten better with a very small / tight claude.md / llm.md file that challenges exactly that

then I'll have another model critique the work at the end of the day

Reply on news.ycombinator.com