Hacker News Reader

Comment by e1g | Hacker News Reader

The paper includes several examples of their modified questions. There has been a substantial jump from o1-preview to o1, so I gave several samples to o1 and o1-pro (not o1-preview), and current o1s gave the correct answer to those modified problems. SOTA changes fast.

View on HN

The paper mentions that on several occasions the LLM will provide a correct answer but will either take big jumps without justifying them or will take illogical steps but end up with the right solution at the end. Did you check for that?

No, I don't know enough math to test the logic, only the check questions against their expected answers in https://anonymous.4open.science/r/putnam-axiom-B57C/data/Put...

Putnam problems need to actually be graded, often the answer itself is trivial.

LLM boosters are so tiresome. You hardly did a rigorous evaluation, the set has been public since October and could have easily been added to the training data.

Your points would be more convincing if you didn’t preface them with arrogant cynicism.

I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check.

Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test.

Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.

> "...roughly on par with a mediocre, but not completely incompetent graduate student"

Let it sink in how vague and almost meaningless that statement is.

What types of questions are you hoping to answer for that to be considered a vague statement?

Reply on news.ycombinator.com