Christopher Chabris Should Calm Down
slate.com
slate.com
"I was simply saying that all writing about social science need not be presented with the formality and precision of the academic world. There is a place for storytelling, in all of its messiness."
Chabris didn't say that Gladwell was informal or merely imprecise. He said he cited studies that have been proven wrong in the service of a story.
In other words, Gladwell did not merely present a scientific conclusion in an informal, lightly supported manner. He presented false conclusions as if they were true, invoking the authority of "science", because it told a better story.
That is an abuse of science and very worthy of scientists being a little "uncalm" about it.
> Chabris does not like this study. He thinks it involved too few subjects and that its findings were not replicated by a subsequent study. Alter and Oppenheimer disagree. They say that the version of desirable difficulty that they explore has been confirmed on numerous other occasions. This is the kind of intramural argument about the nature and value of evidence that social scientists have all the time.
So who am I supposed to trust here? Who has the higher epistemic ground, Chabris or Alter & Oppenheimer? The problem is that I don't know what the scientific consensus is on the validity of that study. I'm not sure whose side I should be taking.
Are Alter & Oppenheimer bad scientists trying to pass off a poor study as robust scholarship? Or is there a valid difference of opinion here?
The problem here is Gladwell didn't even hint that there is significant doubt about the methodology of study he cited. Note his dismissive term for it: a "intramural argument". Chabris just "doesn't like" the study, as if it's a shirt in a color that doesn't suit him.
These are not equivalent things. It's not just a difference of opinion or who tells a better story or who has more letters after their name. What matters is the science behind it, and Gladwell is intentionally perverting it in the service of a better story. That's a huge disservice to the public when it comes to popularizing science.
But they're not the ones who can decide whether subsequent studies replicated their findings or not! It's up to the scientific community as a whole to decide.
The problem is, no one is really interested in replicating earlier findings, and even less in invalidating them. What people want is produce new and interesting (unexpected, curious) results that gets the authors noticed.
Many such studies never get replicated (see the famous one about how talking to students about old age makes them walk slower), and nobody cares.
So this part of Gladwell's response is even more disingenuous than the rest.
Yes, Gladwell mentions Alter and Oppenheimer in his writing, but remember that the knock against Gladwell is that he selectively brandishes science when it suits his narrative. He's just doing that again here.
The fundamental complaint from Chabris is that Gladwell did not even acknowledge the existence of any controversy on this subject in his book. He simply picked the study he liked best and wrote about it as if it were fact.
That's a substantial overstatement. Gladwell didn't cite studies that have been proven wrong, he cited studies that have not been consistently reproducible. That's still a problem, but it's a very significantly different one.
And yet I find his responses very convincing, especially this one and the response to the Ask A Korean! article. I don't know, maybe I'm just easily swayed or maybe it speaks more about how persuasive Gladwell is.
Yea, I had just graduated from college and was binging on popsci. I felt really smart because all the popsci authors seemed to reference each other and when I saw the references I thought I was very well read. Obviously, I don't think that anymore.
This is an embarrassingly accurate description of me at that time: http://xkcd.com/863/
Do other people's comments and criticisms make you change your mind about something that easily? If you thought it was a good book when you read it, why would anything anyone else says change that? (Short of, say, Gladwell made up the entire thing or has been rampantly plagiarizing without attribution.)
Gladwell's books are very well-written narrative combining summarized research studies, anecdotal stories and histories to support his overall thesis. It is not a scientific study. He "connects the dots" of what he believes to be the central theme of his book. I haven't seen anywhere that he "lied".
There will always be detractors to any non-fiction author, especially one so popular. (See Nassim Taleb's books as another example.) If you can find a non-fiction book without major detractors, it probably isn't very good.
It's an interesting illustration of his "narratives often begin in one place and end in another" that his Slate piece begins with calling his detractors ill-read and careless ("I didn’t want to debate them. I just wanted them to read my book all the way to the end."), and ends with the implicit argument that criticizing his work is the same thing as calling his adherents stupid ("I was simply saying that all writing about social science need not be presented with the formality and precision of the academic world...My point was that the people who read my books appreciate this.").
That's a nice example of the sort of false dichotomy that infuriates Gladwell's critics.
Genuine question
I'm not saying Gladwell is right, but Chabris' argument amounts to pulling one referenced study out of context and saying "SEE HOW TERRIBLE THIS IS." He's complaining about sample size... with a single anecdotal example?
Also, social scientists are kind of cute when they get all angry.
Some day, someone's grandkid is going to be spouting something about "10,000 hours" and not even know where it comes from.
Which will replace the even more flawed understanding of "born with it".
We all go through life with flawed mental models, which aren't perfect but are better than a complete lack of understanding. Over time, we make those models in the general consciousness better, even if they continue to lag behind true scientific understanding.
I haven't read a Gladwell book, let alone bothered to run down all the relevant science. But if Gladwell's legion of critics are right, then he's doing the opposite of Mythbusters. They are teaching people to experiment; he's turning ideas into appealing stories without any interest in testing them.
The citations aren't there to trick people into thinking things are more serious than they are. They are meant as foundation and support to the theses of the essays. What more could you ask for? Seems ad hominem.
[1] An example that Chabris points out: "The Law of the Few."
The average reader may not, but at least they've been compelled to pick up a book, read it, synthesize it and draw their own conclusions - either agreeing or otherwise. That's better than nothing.
An above-average reader (whatever that is) might read it and be annoyed. A below-average reader might read it and either not understand it and take it as gospel.
What's the problem? Truly.
I suspect it's much more common that readers believe they have received a reasonable review of the topic. Gladwell's critics are saying that he presents his work as such, but it is not. There's the problem.
I think people often aren't synthesizing and drawing their own conclusions (it is generous to apply this IMHO uncommon quality to the average departure-lounge-pick-up reader). They are reading Gladwell like a textbook.
> My point was that the people who read my books appreciate this. They are perfectly aware of the strengths and weakness of the narrative form. They know what a story can and can’t do, and they understand that narratives sometimes begin in one place and end in another.
The point is that they don't and they aren't and they can't - many people believe that Gladwell is more than a storyteller, and take his scientific gallavantry as truth. It's irresponsible to ignore that fact, but it's necessary for Gladwell's business model to do so - people read his works primarily not because he is a good storyteller, but because they believe he is telling the truth, as backed up by science.
1) He emphasizes that Chabris has written <bold>three</bold> critiques of him as of date and will probably write a fourth
2) He calls the request for debate silly, making Chabris look agressive.
3) Notes that "I clearly drive Chabris crazy"
Everybody loses when the conversation turns personal.
Just saying "Hey, what if?" is not an incredible benefit to society if the claims are not properly testable.
Personally, I never want to stop saying, "What if". That question has led to amazing things.
This is why popularizers and journalists have an incredible responsibility to provide this deconstruction and fact checking on behalf of the public, and it's why there is so much outrage, particularly from the scientific community, when they fail.
Writers should not be blaming their readers for misconceptions and misunderstandings.
> The first striking thing about all three of Chabris’ reviews of David and Goliath is how much attention he pays to a study that I mention at the beginning of my chapter on dyslexia.
Many of the readers of pop-science books do not know how to properly read research papers. They don't have access to the libraries; they don't have the stats knowledge; they don't know how to pick out the decent studies from the poor studies.
It'd be good if pop-sci writers were doing this work for the readers, or at least helping readers to do the work.
"In my review, I criticized Gladwell for describing this experiment at length without also mentioning that a replication attempt with a much larger and more representative sample of subjects did not find an advantage for difficult typefaces. One of the original study's authors wrote to me to argue that his effect is robust when the test questions are at an appropriate level of difficulty for the participants in the experiment, and that his effect has in fact been replicated “conceptually” by other researchers. However, I cannot find any successful direct replications—repetitions of the experiment that use the same methods and get the same results—and direct replication is the evidence that I believe is most relevant."
I'm not sure who to believe here. Legitimate criticism exists, but for every successful author, there is one who seeks to piggy back on their fame by attacking it, and for every sensational scientific study, there is more to be gained by disproving it than by replicating it.
"The kinds of people who read books in America seem to have no problem with my writing. But I am clearly a bee in the bonnet of some of the kinds of people who review books in America."
Which I interpret as: Because I have high sales figures, that clearly proves that experts are easily upset fussbudgets who you shouldn't listen to.