I wonder how many questions give a gentle nudge towards the answer like this. How many answers would have been wildly off the mark without specifying what the answer needs to look like?
> What color is the ball hidden behind the flowerpot in my neighbor's backyard?
Maybe you can reason towards the answer if you only have a deep knowledge of bird anatomy and not Apodiformes anatomy, and that's the intelligence part?
This is testing “knowledge”, not intelligence. And with access to most of the knowledge in the world and basically infinite memory, that’s not very exciting for an AI.
You should be able to easily accept answers like "four" and "4" as equivalent, for example. I doubt there will be that many frontier models running against this test at any time, and a simple glance at the answers from any human should be enough to catch edge cases like this one.
Yeah, but i bet many humans would answer a question like this by naming the number and then listing the tendons. Just writing down a single number to a question worded like that (without the last sentence) feels wrong.
The question goes into too many details and explains too much for that. People mirror the style of the question with their answer. They asked a mini essay so i will answer with a mini essay of my own. If they would have wrote “Number of tendon pairs attached to sesamoid bone of hummingbirds?” Then i would write a single number, no explanation.