If a system could perform generally well, then the implications are astounding. It could imply general analogizing capability. It would also mean AI came way faster than anyone thought.
(And all that's assuming something like g even exists.)
if you didnt know what the test was, then this procedure of using a "specific" algorithm is not distinguishable from general intelligence!
To sum up, the paper should be titled "Dictionary is better than MechTurk users on look-in-dictionary tasks."
Link to the original paper: http://arxiv.org/pdf/1505.07909v1.pdf
> If a system could perform generally well, then the implications are astounding. It could imply general analogizing capability.
There is no reason to expect this at all. IQ tests are designed to measure $g$, which is a strong, profoundly important, but imperfect correlation among humans in how they perform across many cognitive tasks. The fact that all these different tasks (even reaction time!) are correlated for humans is weak evidence that they are all influenced by a single biological factor. However, there is this very little reason to think that all these tasks will be correlated for arbitrary reasoning systems, especially a system specifically trained to perform on one specific test.
An analogy: the lengths of various body parts in humans (finger length, leg length, etc.) are all very correlated, and point to a general factor of size, i.e., it is reasonable to talk about "tall" humans and "short" humans, which generally have all their body parts longer or shorter, respectively. But if we're designing humanoid robots from scratch, there is no reason to expect there to be such a general factor of size. Some robots will be designed for welding car doors, and will have arms but no legs!
In short, the fact that IQ tests measure g, which is then predictive across many cognitive tasks, is crucial based on the fact that the sample set is humans, for which a small number of causal mechanisms (genetics, nutrician, etc.) are responsible for the correlation.
With a little prodding and tweaking, I can imagine a search engine doing decent on many classes of standardized tests; and so can test administrators, leading (partially) to why smart-phones/computers/connected-devices are generally forbidden when taking standardized tests.
Nobody agrees what Intelligence is, and people with a high IQ are not always clever.