Hacker News Reader

Comment by brap | Hacker News Reader

Where's the comparison with Gemini 2.5 Pro?

For coding, I like the Aider polyglot benchmark, since it covers multiple programming languages.

Gemini 2.5 Pro got 72.9%

o3 high gets 81.3%, o4-mini high gets 68.9%

thanks

Isn't it easy to train on the specific Exercism exercises that this benchmark uses?

It was a good benchmark until it entered the training set.

Exactly.