> seem to be very one dimensional in terms of measuring intelligence.
I would argue that they DON'T measure intelligence, rather they test knowledge.
Frustratingly, I think we have a society greatly focused on knowledge based testing due to its correlation with intelligence and that it is exponentially easier to test knowledge. But this is easy to hack. Being in CS it feels very odd since we all know a great way to get hired is to study leetcode questions. That is, study to the test.
This is critical to recognize this difference as what we know for certain is that LLMs and other ML systems are analogous to a database with a human language interface[0]. What we DO NOT KNOW is if these systems are intelligent. That is, that they can use their exploit their knowledge to unfamiliar territories. Then there's the whole question of wisdom...
This stuff is highly abstract and we can get fuzzy so it is natural to go for the simple thing but we need to graduate. Don't avoid the tough questions, dig in. As we advance in any study nuance takes over. This should be obvious. If we approximate things, to improve we need to tackle higher order terms, and that almost always becomes exponentially more difficult with each step.
And come on, is this benchmark not obvious bait? Calling it "humanity's last exam" is extremely arrogant.
Definitions:
Knowledge: Awareness of facts. The ability to recall information.
Intelligence: Ability to exploit knowledge to new settings. To be able to plan and reason.
(Definitions of intelligence are much more debated than knowledge but what is far less controversial is that intelligence is about the way one uses knowledge. These two are distinct. This is fairly well agreed upon throughout history and within modern literature around psychology and cognitive science.)
Wisdom: The efficient use of one's knowledge
https://en.wikipedia.org/wiki/Knowledge
https://en.wikipedia.org/wiki/Intelligence
https://en.wikipedia.org/wiki/Wisdom
There is a implicit hierarchy here[1] where knowledge is something to be had, intelligence is the utilization of that, and wisdom is about efficiency. There's a decent analogy to this hierarchy. Knowledge is like having a tool. Intelligence is like using it, a craftsman[2]. Wisdom is akin to being a master craftsman.
[0] I mean that they fit the data. A database is discrete, but these curve fit, so that will be a continuous function (in most cases). Thus it won't be exact retrieval nor does this mean information can't be interpolated. But that gets to be a deeper and much more complex conversation that I think we like to admit.
[1] This is clearly multi-dimensional. You can organize hierarchies in multiple ways, I'm not suggesting this is the only way or "the right way"
[2] What is argued is what is a sufficient threshold. An armchair expert might know how to use a lathe because they read about its usage but does that mean they can use it? What about a novice who you can show something to and they can repeat it? Monkey see monkey do style. An apprentice? A craftsman? There's a lot of gray area between being able to recall something from a book and being a wizard (gray beard).