For example, the WAIS-IV measures "working memory", in part, via orally administered arithmetic word problems. An imminently reasonable thing to associate with intelligence in the pre-pocket-calculator economy!
But today? Kids spend a lot less time drilling arithmetic now than N decades ago. Those drills are more often administered on computer screens than orally in front of a classroom. And almost no one does a non-trivial amount of mental arithmetic in their work life. So this is mostly a useless skill. And, because it's a useless skill, we devote a lot less time to mastery. Even once mastered, the skill atrophies because it's just not very useful.
Question: if scores on these oral arithmetic word problems only barely decline despite significantly less practice -- both in the classroom and in everyday life -- what should we conclude?
Stated differently: if someone with 10K hours of tetris grind drops 2% more blocks than a total newbie, who's likely to be more adept at other forms of spatial reasoning?
The more general critique is that "intelligence research" has ossified around Cargo Cult psychometrics. At one point, "orally administered arithmetic word problems" were a fantastic proxy for economically useful mental faculties. That is no longer true. As the set of useful capacities change, so too should psychometric evaluations. Both because treatment effects are going to make cross-generational comparisons worse than useless, and also because the thing being measured has become irrelevant.
The humble pocket calculator should have taken Sociology by storm half a century ago. But the field hasn't kept up, and the problem is about to get exponentially worse as generative AI tools and other types of advanced automation start to augment every aspect of mental labor.
What it means to be "good at solving hard problems and getting things done" when assisted by GPT-614830 might look entirely different from what it means to be "good at solving hard problems and getting things done" today.