Humanity's Last Exam

lastexam.ai

15 pointschvid3 comments

> In Greek mythology, who was Jason's maternal great-grandfather?

These tests are getting ridiculous (especially when you take into account the name they choose for them...).

For me things are quite simple. I witness the limitations of LLMs every day. What this benchmark checks if what data it was trained on, that's pretty much it. No need to get "humanity" involved into this.

To me, comments like yours prove that goal posts are constantly moved every day.

But there's also been substantially more investment now than say 2 years ago (AI + AI-adjecent companies have, what, 4-5 trillion USD valuation now?). CEOs tout how they stopped hiring. And then these benchmarks having names like "humanity's last exam". All that implies LLMs are a direct replacement for human workers so imo it's fair to measure LLMs against the hype/promises.

So I would say the goalposts are moving just in step with the market expectations/hype. Nobody's disputing that LLMs are amazing at generating useful text anymore.

Reply on news.ycombinator.com