Hacker News Reader

Comment by pron | Hacker News Reader

Small sample sizes and publication bias are a lethal combination in any field.

Suppose scientists wanted to test the hypothesis that a fair coin always comes up heads. Due to budgetary issues, scientists are only able to toss the coin 5 times. 40 groups of researchers conduct the experiment in universities around the world. One of them is quite likely to get 5 heads. Guess which result is likely to be published (or which group is going to even attempt to publish its result). Moreover, this result is statistically significant according to the well-accepted peer-reviewed journal standard of p < 0.05.

I suggest people read John Ioannidis's paper: Why Most Published Research Findings Are False (http://www.plosmedicine.org/article/info:doi/10.1371/journal...), and the Atlantic story about Ioannidis and his work: Lies, Damned Lies, and Medical Science (http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...)

View on HN

Here is a more complete reading list:

Lies, Damned Lies, and Medical Science (http://www.theatlantic.com/magazine/print/2010/11/lies-damne...)

The Truth Wears Off (http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_...)

Grant System Leads Cancer Researchers to Play It Safe (http://www.nytimes.com/2009/06/28/health/research/28cancer.h...)

Placebos Are Getting More Effective. Drugmakers Are Desperate to Know Why. (http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo...)

The Cult of Pharmacology: How America Became the World's Most Troubled Drug Culture (http://www.erowid.org/library/review/review.php?p=269)

The Mouse Trap: The dangers of using one lab animal to study every disease (http://www.slate.com/articles/health_and_science/the_mouse_t...)

Why Most Published Research Findings Are False (http://www.plosmedicine.org/article/info:doi/10.1371/journal...)

Mistakes in Scientific Studies Surge (http://online.wsj.com/article/SB1000142405270230362710457641...)

Trials and Errors: Why Science Is Failing Us (http://www.wired.com/magazine/2011/12/ff_causation/all/1)

In cancer science, many "discoveries" don't hold up (http://www.reuters.com/article/2012/03/28/us-science-cancer-...)

Lullaby for a Mouse: Anthropological Observations of an Animal Model of Psychosis (http://vimeo.com/33803995)

Inside the Battle to Define Mental Illness (http://www.wired.com/magazine/2010/12/ff_dsmv/all/1)

There are of course quite a few relevant books as well, but that's probably enough to keep most people busy for a few days. Also, most of these articles are about drug research, medicine, or psychology, but it applies to basically every field.

I suggest you stay away from "The Truth Wears Off" by Jonah Lehrer. It's filled with errors and exaggerations, and it seems like Lehrer's concern was demonstrating the existence of a Gladwell-like bogus "phenomenon" which he calls the "decline effect".

It is a well written popular article about his own experiences. A candid honesty about his own failings. The decline effect is humour (and a catchy tagline) rather than attempting to be a testable hypothesis.

Interesting, is there any discussion of this online?

http://scienceblogs.com/insolence/2010/12/is_the_decline_eff...

http://scienceblogs.com/insolence/2011/01/is_the_decline_eff...

http://theness.com/neurologicablog/index.php/the-decline-eff...

http://scienceblogs.com/pharyngula/2010/12/science_is_not_de...

http://www.talyarkoni.org/blog/2010/12/15/the-decline-effect...

Also, there used to be a lively discussion in the comments to this blog post (http://www.newyorker.com/online/blogs/newsdesk/2011/01/jonah...) by Lehrer, but all the comments seem to be gone now.

I would add to the list: http://thelastpsychiatrist.com/2011/02/the_decline_effect_is...

The main point: Psychology is not "science" in the same way F=ma is science. It's a model.

The secondary point:

  When the article then goes on to describe the breakdown of this 
  sweeping generalization in studies after 1994 (on other species), 
  it attributes that to the Decline Effect.  It's not.  When you 
  look at the studies together, what you should have inferred is 
  "symmetry is an associated  factor in mate selection by females 
  in only some species and not others and more research is need to 
  explain why."  Instead, the article attributes its inability to 
  summarize the variety and complexity of nature in a 140 character 
  Twitter message to an underlying failure in the 500-year-old 
  guiding principle of science."

I don't disagree with you that Psychology is not science, but F=ma is a model as well.

I would add "The Heart of Research is Sick" to that list. The problems it outlines closely match my own experience.

http://www.labtimes.org/labtimes/issues/lt2011/lt02/lt_2011_...

Thanks! (And also thanks to pron, boredguy8, and duwease.)

I have an enormous mindmap/document of all the most important facts and reading for understanding various major areas of society, so I am going to version all of this stuff in once I get through it. Eventually this entire document will be published, but this will take a while.

I suggest the book "Wrong: Why experts* keep failing us--and how to know when not to trust them" by David H. Freedman, the author of that Atlantic piece. It's a book length treatment, and very damning in many ways.

Reply on news.ycombinator.com