We're gonna need some new benchmarks...
ARC-AGI-3 might be the only remaining benchmark below 50%
ARC-AGI-3 might be the only remaining benchmark below 50%
GPT 5.4 Pro leads Frontier Maths Tier 4 at 35%: https://epoch.ai/benchmarks/frontiermath-tier-4/
You can't consistently benchmark something that is qualitative by nature. I'm struggling to understand how people don't understand this.
Here is an example question: https://i.redd.it/5jl000p9csee1.jpeg
No human could even score 5% on HLE.
That is, it's easy to make benchmarks which humans are bad at, humans are really bad at many things.
Divide 123094382345234523452345111 by 0.1234243131324, guess what, humans would find that hard, computers easy. But it doesn't mean much.
Humanity's last exam (HLE) couldn't be completed by most of humanity, the vast majority, so it doesn't really capture anything about humanity or mean much if a computer can do it.