The odd one out - inspired by IQ tests
blog.tanyakhovanova.com
blog.tanyakhovanova.com
Some reasons are obvious: The test might be intentionally designed to measure something other than IQ (success in college, performance on a job, money spent on College Board® prep courses). The test might be designed to test within a particular IQ range, whether high or low. The test might be designed only to be taken by people fluent in a particular language, or who received an education from a particular school. And so on.
But the biggest and least obvious reason why bad IQ tests are everywhere is that good IQ tests are not only difficult to make, but expensive to make. Good tests are not made by geniuses in armchairs thinking up riddles. Test design is an empirical science, and the final judges of any question's value are the statistical relationships that show up in the answers when the test is given.
Empirical evidence sterilizes tests from the biases of their authors, and does so much more effectively than waiting for complaints from morally outraged observers. Questions that penalize takers of different cultural backgrounds are bad questions not because they're socially repugnant, but because they're noisy--you can demonstrate that people who otherwise score highly get the question wrong more often than they should. Questions like those addressed in the blog post, where some really smart people are attracted to a "wrong" answer, will have negative inter-question correlations among the brightest subjects, which is a giant blinking neon sign in any analysis.
This sanitizing effect is so strong that the tests which show the most desirable psychometric properties, such as Raven's Progressive Matrices, often evoke the opposite complaint: they're so minimal and homogenous in form that it's hard to accept they're a better measure of IQ than scattershot tests of imperfect analogies and odd-one-out puzzles. If you're curious, RPM goes a little something like this: http://iqtest.dk/main.swf
IQ testing is a complicated science, and discrediting it because there's one or two (or fifty) dumb questions on the SATs is like giving up on journalism after reading an issue of USA Today.
I'm not against testing. There has to be some way to decide who gets into what college, and the SAT is a published standard. But there's no point in defending a flawed system. Let's just accept that IQ tests are a bit ridiculous and move on.
Well, you could, but first you must define it! ;)
The interesting part is that, even defined in that rather circular manner, IQ can be consistently measured and compared between individuals and has strong, reliable correlations with performance on other tasks commonly seen as intelligence-based, as well as correlating with both academic and professional success. Yes, these are correlations, not guarantees, but don't underestimate the value of that, and it's not actually clear that there's any better way.
The unknown quantity that such tests are trying to measure is usually referred to as g, or the general intelligence factor. Look up the term "g-loaded" for more information.
Do you actually believe this dogma?
This Usenet post should be read by all "IQ deniers": http://tiny.cc/F7gxA
"according to IQ doubters, knowing how smart I am should tell you nothing about my IQ. It is as likely to be 50 as 150."
The same poster once proposed a beautiful test of the depth of an IQ denier's beliefs:
You are to be operated on, and have been given a choice between two surgeons. You know nothing about them, save for the IQ of each. Will you choose the surgeon with the lower IQ? What if the difference between them is an entire standard deviation? Two standard deviations? If you truly believe that IQ tests measure nothing - that the result of any such test is essentially a random number - you should happily choose the surgeon with the lower score. If you do not believe it, we are merely quibbling over the accuracy of existing tests.
Moreover, the only way to seriously dismiss the existing IQ tests as hopelessly inaccurate is to borrow tricks from the Creationists' playbook wholesale (as Steven Jay Gould, the patron saint of IQ denial, often did) and close your eyes (and more importantly, those of your audience) to reams of high-quality data.
More references, for those actually interested: http://www.psych.utoronto.ca/users/reingold/courses/intellig...
And if you want to know the core driving force behind the IQ denialists:
http://www.gnxp.com/blog/2007/10/james-watson-tells-inconven...
"Watson was thinking like a scientist. Which is exactly why he was punished. The moral laws of our society dictate that we are not allowed to think scientifically about some issues. Especially not in public."
Investors don't seem to care about IQ at all. It wasn't an issue of public consideration during the last presidential election. If it could reliably predict anything, don't you think more people would ask about it? I'm sure there are places where IQ scores are considered (applying to college or Google), but these are the exception, not the rule.
There is generally a difference between people with an IQ of 50 and an IQ of 150. But you don't need a standardized test to see those differences. The problem is that there is very little difference between an IQ of 120 and an IQ of 130. If all IQ is good for is telling the difference between the very intelligent and the learning disabled, then it isn't very useful in daily life.
It absolutely is used. We are simply forced to measure it through indirect means, given that IQ tests are verboten.
> Does anyone even know their surgeon's IQ?
Medical school is (perhaps above all) a proxy IQ test.
> Investors don't seem to care about IQ at all.
They are quite interested in heavily IQ-correlated traits.
> If it could reliably predict anything, don't you think more people would ask about it?
U.S. law de-facto prohibits the use of IQ tests for anything of economic importance: http://caselaw.lp.findlaw.com/scripts/getcase.pl?court=US...
> But you don't need a standardized test to see those differences.
There is also no need to use a thermometer in laboratory work - we're all equipped with fingers. At least ten measurements' worth, at that.
Indirect measures are useful.
Want to determine which teenagers would make decent electrical engineers? Instead of $500 worth of IQ test (proper IQ tests are administered by private psychologists, who do not work for free) one can use $150,000 worth of college/trial-by-ordeal - a poorer measure of general intelligence, overall. Yet it continues to be the only allowed measure, because IQ tests are Officially Evil.
> there is very little difference between an IQ of 120 and an IQ of 130
On what basis do you say this?
> If all IQ is good for is telling the difference between the very intelligent and the learning disabled
This is merely its last permitted use.
Intelligence should not by measured by a standardized test. Standardized tests simply do not have a history of predicting success at anything on a micro level. If they did, people would use them. It would be trivial for large employers to pay $500 per employee to have an IQ test taken. And it isn't illegal for them to do so. From the court decision:
"Nothing in the Act precludes the use of testing or measuring procedures; obviously they are useful. What Congress has forbidden is giving these devices and mechanisms controlling force unless they are demonstrably a reasonable measure of job performance."
The law doesn't say that you can't use IQ tests for anything of economic importance. It says that you can't use them to discriminate. Do you think Google could show that a high score on a general intelligence test bears a demonstrable relationship to successful performance as a software engineer? And if it doesn't...then what good are the test scores?
It is possible that there is a great conspiracy against IQ tests. They could be the best measurement of intelligence, but people just don't like them. It's also very possible that the test scores aren't used in society because are relatively meaningless.
This is empirically false. Read about the Longitudinal Study of Youth. Children took tests, grew up, lived their lives. You cannot argue the data out of existence.
> The law doesn't say that you can't use IQ tests for anything of economic importance. It says that you can't use them to discriminate.
There are many practices which are not illegal per se, but put you at dire financial risk liability-wise.
Read about the actual consequences of the precedent set by the Griggs v. Duke Power Co. decision. Talk to a lawyer about whether it can ever be safe to include a traditional IQ test in your company's hiring process. The problems involved in demonstrating "non-discriminatory intent" and "performance applicability" are insurmountable. Unless you have extraordinarily deep pockets, the sheer expense (not to mention PR debacle) of fighting such a lawsuit will sink you before you are at any risk of overturning the precedent.
> It's also very possible that the test scores aren't used in society because are relatively meaningless.
Once again, the falsehood of this statement can be uncovered by anyone who cares to look. We aren't talking about an organized conspiracy here - only a generation's worth of toxic political correctness.
Features
========
Color Frame Size Shape
----- ----- ---- -----
1. red yes large square
2. red no large square
3. red yes large circle
4. green yes large square
5. red yes small square
Features in Common
==================
1 2 3 4 5
1 - 3 3 3 3
2 3 - 2 2 2
3 3 2 - 2 2
4 3 2 2 - 2
5 3 2 2 2 -"What comes next in this sequence? 1,4,9,16,25,"
Sure, the answer might be 36, but then again maybe we're looking at some subtler sequence of numbers than that. Maybe it's "the number of bald men who walked past my house every hour since 5am". Maybe the next number is 345 due to the annual Patrick Stewart Lookalike Parade. Who knows?
Oh, certainly. I've never found a case where there's genuine ambiguity, it's just that there's a certain part of my brain which takes delight in pointing out all the potential flaws in everything I read. It's much harder to concentrate on the test when half my brain is busy visualizing what four hundred Captain Picards would look like marching down my street.
I recall a test that asked to pick the one word out of four that was different or didn't fit or something.
The thing was, three out of four words described red objects. But a different three out of four words had one syllable; the other had two.
So it them became a game of, What was the test creator likely thinking?
I don't know if IQ tests tell anything about a person's real intelligence, but I think that most intelligent people can become very good at solving these tests.
This is why proofs using K-complexity (forgive me, I often misspell it) always use it in a way that doesn't involve actually assigning numbers to the K-complexity; while it has certain desirable properties, there is no unique K-complexity value for a given language.
Aaaaand it's exactly this reason I also loathe those questions. It is much less "do this mathematical thing" than "read the puzzle-maker's mind", and quite frequently there is simply literally not enough information in the sequence to read the puzzle-makers mind. With an uncountably infinite number of functions to choose from and a finite set of inputs to choose among them, it's a complete joke of a test. On the other hand, with suitable constraints given in advance it could prove quite useful. (... but that would remove the thrill of lording the answer over people, methinks, which I think is a distressingly large part of such problem's appeal....)
in the limit, all schemes are roughly equivalent. (they are equivalent up to constants.)
> given the nature of the problems in question that's going to really not work
ok. so attempting to compute the kolmogorov complexity for iq test problems is a bad idea. it isnt actually a computable function. its existence shows there is a mathematically rigorous way to talk about the most plausible way to extend an integer sequence. more than anything, this is a counter to the argument that "an uncountably infinite number of functions to choose from and a finite set of inputs to choose among them" makes the test entirely subjective.
> there is no unique K-complexity value for a given language.
for any given language, there is, unless i dont understand what youre saying.
> your encoding scheme favors polynomials but makes expression squares a royal pain
i know its wrong, but i cant help pointing out that that doesnt actually make any sense at all
Be careful what you are testing.
For example, in dim light, I can find it difficult to tell the difference between brown and red on a snooker table. But close up, or in bright lighting, there's no problem. For example, there's a red, a green and a brown ball in this picture:
http://upload.wikimedia.org/wikipedia/commons/e/e6/Snooker_t...
I can tell the difference between the red and the green more easily than the red and the brown. If the red wasn't visible on the table for comparison, and the brown wasn't near its home spot on the centre of the baulk line, I wouldn't be sure it's not red.
I doubt that 10% of males have the same level of difficulty distinguishing red from green as you indicate.
If that is an available answer, then the question can effectively separate subjective, presumptuous approaches from objective, analytical ones.
If "it depends..." is not an available choice, then it demonstrates that the test itself is subjective and presumptuous, but this doesn't necessarily implicate standardized tests in general.
It just demonstrates that this question might be a good one, if what you're testing for is the order that people find "the odd one" (not necessarily in an IQ test).
The next reason states it more clearly imo.
2) the first symbol differs from the other four symbols in exactly 1 way; the other symbols differ from each other in 2 ways.
On a typical test, when given a similar problem, I would have to try and figure out the thinking style of the test designers.