For the types of testing that he's doing, I suspect he's measuring boredom more than anything else, especially since he's testing largely in a university setting. Intelligent people are accustomed to being bored with endless entry-level evaluation exams, and at first glance this looks like it's just one more of them. And because the stakes here are so low (essentially zero), lots of people will just fly through without really reading and analyzing the question.

What he's seeing isn't something new, it's something so old that it's part of popular culture: the absent-minded professor syndrome. It's the stereotype of the brilliant physicist forgets what he's supposed to buy at the supermarket because he's thinking about their quantum properties. Analytic people are horrible at things that don't interest them.

Pay the students $50 for each correct answer, and there's not a doubt in my mind that the results will be the complete opposite of what he's seeing now.