Hacker News Reader

Comment by WiSaGaN | Hacker News Reader

I don't think this proves that the LLM is just "pattern matcher". Human makes similar mistakes too, especially when under time pressure (similar to non-reasoning model that needs to "use system one" to generate answer on one go). This is further evident that if you specifically ask the models to pay attention to traps, or just ask follow up question "are you sure?", then they usually can get it right.

View on HN

You're saying that humans perform worse on problems that are slightly different than previously published forms of the same problem? To be clear we are only talking about changing variable names and constants here.

Often yes, because we assume we already know the answer and jump to the conclusion. At least those of us with ADHD do.

Not really true for Putnam problems since you have to write a proof. You literally can’t just jump to a conclusion and succeed.

That is the principle behind the game 'Simon says'

'Simon says' is about reaction time and pressure.

No, it’s not at all.

This is all getting so tiresome.

That’s a very silly analogy. A more realistic analogy would be do humans perform better on computing 37x41 or 87x91 (with showing the work)?

It was not an analogy at all. It was an simplified example of the idea that a slight change in a pattern can induce error in humans.

It seems some people disagree that that is what the game "Simon Says" is about. I feel like they might play a vastly simplified version of the game that I am familiar with.

There was a recent episode of Game Changer based on this which is an excellent example of how the game leader should attempt to induce errors by making a change that does not get correctly accounted for.

Reply on news.ycombinator.com