The Implicit Association Test falls short of basic scientific standards

Being hesitant to uncritically embrace iffy research that supports our politics can be really hard. In myself I even notice a tendency when looking at research on controversial subjects to put more weight on articles that I "like" going in, and I will sometimes go as far as to interpret articles that disagree with my views as partially supporting them, if read in a certain way.

The thing is, it's 100% correct to evaluate new claims in light of your priors. That's absolutely necessary if you want to make good decisions - my expectation that special relativity will not be shown to be inconsistent is so strong that no matter how interesting or new an argument is against it, I will shred it to bits way more viciously than I'd pick apart a simple replication of special relativity's claims.

But.

Some priors don't come from reliable places, and we have to be way more careful about applying them. In general, the more politically or socially relevant an issue is (gender, race, money, morals) the less any of us should trust both a) our own priors, and b) the priors of researchers in the area, who have many ways to unwittingly and without bad intention put fingers on the scale. In those situations we really have to throw our hands up and wait for meta-analyses to tell us what's up.

In this case, it's super worth noting that even if the IAT is a garbage test, that doesn't mean that implicit bias doesn't exist, or that racism is not a problem. It just means the test doesn't measure it, and we shouldn't be funding further research that leans on it as a tool. Which is why I really don't understand why criticisms of the test draw such intense opposition.

I'd like to say that "everyone already knows this", but this particular topic is one that seems to be a poster child for people believing what fits their politics instead of what fits reality, unfortunately. So it's always good to see it getting more exposure.

I honestly don't get what's in it for the people pushing bad science related to social-justice issues (excluding those who are actually part of the industry of seminars around same). This is a problem that many people want to solve, and the bad science just makes it _harder_ to solve these problems, and lessens support for the movement to do so.

I would be pretty amazed if the test didn't indicate bias between a complete A-A test (i.e. associate white people and white people with red words and green words), even leaving aside consistent scores as they discuss in the article.

Which means it's not reliably going to indicate anything in particular by itself. This is a challenge for a lot of these psychology tests and is part of the root crisis of reproducibility.

If all variants of this test act as the author describes, wouldn't the fact that the same keys are bound to different things in different phases of the test be a problem?

Surely everyone who quotes this study will amend their statements at this point.

>There are various psychological tests purporting to measure implicit bias; the IAT is by far the most widely used. When social psychologists Banaji (now at Harvard University) and Anthony Greenwald of the University of Washington first made the test public almost 20 years ago, the accompanying press release described it as revealing “the roots of” unconscious prejudice in 90-95% of people. It has been promoted as such in the years since then, most vigorously by “Project Implicit,” a nonprofit based at Harvard University and founded by the creators of the test, along with University of Virginia social psychologist Brian Nosek. Project Implicit’s stated aim is to “educate the public about hidden biases”; some 17 million implicit bias tests had been taken online by October 2015, courtesy of the nonprofit.

“Implicit bias” became a buzzword largely thanks to claims that the IAT could measure unconscious prejudice. The IAT itself doesn’t purport to increase diversity or put an end to discriminatory managers. But it has certainly been deployed that way, partly due to its creators’ outreach.

HR departments quickly picked up the theory, and implicit-bias workshops are now relied on by companies hoping to create more egalitarian workplaces. Google, Facebook, and other Silicon Valley giants proudly crow about their implicit-bias trainings. The results are underwhelming, at best. Facebook has made just incremental improvements in diversity; Google insists it’s trying but can’t show real results; and Pinterest found that unconscious bias training simply didn’t make a difference.

Current studies have found the race IAT to have a test-retest reliability score of 0.44, while the IAT overall is around 0.5 (pdf); even the high end of that range is considered “unacceptable” in psychology. It means users get wildly different scores whenever they retake the test.

“For other aspects of psychology if you have a test that’s not replicated at 0.7, 0.8, you just don’t use it,” says Machery.

The second major concern is the IAT’s “validity,” a measure of how effective a test is at gauging what it aims to test. Validity is firmly established by showing that test results can predict related behaviors, and the creators of the IAT have long insisted their test can predict discriminatory behavior. This point is absolutely crucial: after all, if a test claiming to expose unconscious prejudice does not correlate with evidence of prejudice, there’s little reason to take it seriously.

Essentially it all boils down to this statement: >No psychologist or neuroscientist can convincingly point to a clear divide between conscious and unconscious thought. And so psychology’s attempt to solve discrimination by delineating between an amorphous collection of conscious and unconscious biases is both simplistic and misguided.

Pretty amazing.