Hacker News Reader

Hacker News

Comment by blendergeek | Hacker News Reader

Parent Full thread

blendergeek

View on HN

Note that it's two different things:

This OP claims the publicly available models all failed to get Bronze.

OpenAI tweet claims there is an unreleased model that can get Gold.

I'd also be highly wary of the method they used because of statements like this:

>we note that the vast majority of its answers simply stated the final answer without additional justification

While the reasoning steps are obviously important for judging human participant answers, none of the current big-game providers disclose their actual reasoning tokens. So unless they got direct internal access to these models from the big companies (which seems highly unlikely), this might be yet another failed study designed to (of which we have seen several in recent months, even by serious parties).

My (unreleased) cat did even better than the OpenAI model. No you cannot see. Yes you have to trust me. Now gimme more money.

Wow, that’s incredible. Cats are progressing so fast, especially unreleased cats seem to be doing much better. My two orange kitties aren’t doing well on math problems but obviously that’s because I’m not prompting the right way – any day now. If I ever get it to work, I’ll be sure to share the achievements on X, while carefully avoiding explaining how I did it or provide any data that can corroborate the claims.

I don't know the details (of course, it's unreleased), but note that MathArena evaluated "average of 4 attempts", and limited token usages to 64k.

OpenAI likely had unlimited tokens, and evaluated "best of N attempts."

That's a claim that is far less plausible. OpenAI could have thrown more resources at the problem and I would be surprised if that didn't improve the results.

The model did not fit in the margin.

We'll never know how many GPUs and other assistance (like custom code paths) this model got.

Exactly. Whom to believe?

The last time someone claimed a medal in an olympiad like this, turned out they manually translated the problem into Lean and then ran a brute force search algorithm to find a proof. For 60 hours. On a supercomputer.

Meanwhile high schoolers get a piece of paper and 4.5 hours.

Even though chess is now effectively solved against human players, I still remember Kasparov's suspicion that one of Deep Blue's moves had a human touch. It was never proven or disproven, but I trust Kasparov's deep intuition amplified by Kasparov requesting access to Deep Blue’s logs, and IBM refusing to share them in full. For more discussions see [1][2][3].

[1] https://chess.stackexchange.com/questions/9959/did-deep-blue...

[2] https://nautil.us/why-the-chess-computer-deep-blue-played-li...

[3] https://en.chessbase.com/post/deep-blue-s-cheating-move

kinda wild that an llm cant translate to lean?

Both are true. One spent $400 in compute and the other one spent a lot more.

Exactly. And presumably had a more sophisticated harness around the model, longer reasoning chains, best of N, self judging, etc

OpenAI achieved Gold on an unreleased model. GPT-5. Read the tweets and they explain what they did.

Actually, I did it a year ago but I just don't want to release my model.

Where should I address the billion dollar check?

My buddy did it 5 years ago. You wouldn’t know him, he lives in Canada.

my model goes to a different school

The dog ate mine. And the solution didn't fit in the margin, anyway.

OpenAI explicitly said it’s not GPT-5 but another experimental research model https://x.com/alexwei_/status/1946477756738629827?s=46

Thanks. I parsed that wrong. In either case not the same thing Math Arena used.

Reply on news.ycombinator.com