Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

arxiv.org

13 pointsmfiguiere2 comments

> The most frequent failure mode among human participants is the inability to find a correct solution Typically, human participants have a clear sense of whether they solved a problem correctly. In contrast, all evaluated LLMs consistently claimed to have solved the problems.

This is exactly the problem that needs to be solved. The yes-man nature of LLMs is the biggest inhibitor to progress, as a model that cannot self evaluate well cannot learn.

If we solve this though, combined with reasoning, I feel somewhat confident we will be able to achieve “AGI,” at least over text-accessible domains.

I evaluated a lot of Math Olympiads in Argentina. Humans participants many times don't have a clear sense of whether they solved a problem correctly. We get review request of the grades, sometimes it's a mistake but the grader, sometimes it's a tricky error that changes the result very little, sometimes the human participant made a chain of huge mistakes that make no sense. (Obviously, we give a polite reply in every case.)

Reply on news.ycombinator.com