Why you shouldn’t believe that exciting new medical study
vox.com
vox.com
SML1 - the first study on this topic, may be wrong, but could be pioneering
SML2 - the first confirmation study on an SML1 study, confirming the results
SML3 - one of multiple confirmation studies
SML4 - a consensus or meta-study that confirms the original SML1 study
SML5 - scientific consensus indicates the SML1 study is correct based on current scientific knowledge and is unlikely to be refuted in the near term.
Nobody outside of the immediate scientific circle should care at all until a topic is in SML3, and then it should be treated as bleeding edge and only show up in harder to digest science popularization magazines: Sci-Am for example. SML4 would be ready for the Pop-Sci "emerging science" column. And SML5 would be fit for newspapers and the general public.
- cnn has 9 whole mentions, ever - https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
-washington post has 50, https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
shouldn't this just be part of a normal journalistic vetting before publishing process?
I don't think that this lends itself very well to the publishing process of research in general (meant as a stream of single researches).
Cochrane is one of the foundations of the book Bad science, which you would probably find interesting (assuming you're interested in the subject).
When it comes to the media, the situation is complex, because the interests of the media (and the institutions and so on) can be many, and research quality can be not the only one.
If on a scale from 0 to 10 of interest in quality, the Cochrane institution is a (just to say) 10, the Daily Mail would be a -5, since sensationalized garbage is much more productive for them than boring, low-key, truthful research.
[Of course, I don't mean that Daily mail is certainly not the reference or the standard.]
Or just doesn't care, it could also be the revival of yellow journalism.
"Every once in a while, a story comes through 60 Minutes that has the potential to change lives. This week, Scott Pelley's report on a new medical treatment using the polio virus to kill cancer is that kind of story."
http://www.cbsnews.com/news/using-polio-to-kill-cancer-60-mi...
Stop watching TV news. You won't learn anything and it's designed to make you anxious.
The Jeeps were driven by robots that moved the steering wheel twice as fast as a human could, and even then the 4 Jeeps they tested a zillion times were a lot more stable than you would think (I drove a ~1966 Kaiser CJ-5 1977-79). They over pressurized an Audi's automatic transmission, even plugged a pressure relief valve, to demonstrate the failure they were claiming. Alar has never been shown to be a carcinogen, although if you feed near LD50 doses of a breakdown product to lab rodents you can show a carcinogenic effect. If you drink over 5,000 gallons of apple juice a day Alar might be a threat, but that's well over the LD50 (lethal dose for 50% of a tested population) of just plain water, which is around a gallon and a half at once.
http://www.cjr.org/analysis/understanding_science_journalism...
A dialog between two scholars about how make scientific research more trustworthy:
http://www.washingtonpost.com/blogs/monkey-cage/wp/2015/03/1...
A blog post, "How Can We Improve the Article Review and Submission Process?" with many suggestions for improving scientific publishing:
http://scholarlykitchen.sspnet.org/2015/03/26/ask-the-chefs-...
In the end, we have to count on incremental improvements in publishing, and incremental improvements in reading by all of us, to prompt medical studies and all kinds of scientific studies to become more accurate and informative about important issues. We especially have to improve our own tendency to seize a hopeful answer whether or not it's a true answer as we grapple with life's problems. (When I write "we" here I am including all of us, including myself, whether we are producers or consumers of scientific studies.) Medical studies specifically and scientific studies in general have helped humankind gain a lot of useful knowledge, but it has been a slow, unsteady process with lots of back and forth.
P.S. All the links I've posted here, including the one opening the thread, are from the "Weekend Reads" weekly post on the Retraction Watch group blog, which I think many Hacker News participants will find interesting.
http://retractionwatch.com/2015/03/28/weekend-reads-monsanto...
Not a week goes by without a ton of these, revolutionary improvements on battery technology, solar power, wind power, curing cancer, curing AIDS, things that give you cancer and on and on.
There must be money in it, I can't think of any other reason why structurally mis-informing the public is fun or useful.
Think about it...you dedicate your life to your science, and grind day after day, year after year with no verifiable results, because, you know, in case you haven't noticed, curing cancer is motherfucking hard.
I can see where maybe at some point, you just want to feel some sense of accomplishment or to be noticed in your social group, so you find some questionably-causal, but statistically-relevant piece of research you did and just publish the damned thing.
Don't get me wrong, I find this sort of thing appalling, but in the end, most of us are fairly imperfect people.
At least that is what I take away from a lot of it.
What's more interesting then a headline that promises some scientific breakthru is about to change the world?
The real issue, I believe, is that there is no accountability for almost anyone these days. If there is some upside in publishing or reporting on questionable science, and no downside, what is the incentive not to do it?
I fear that no training can be helpful if you're supposed to have 10 stories written by this afternoon, and they'd better be good at attracting readers, or else.
> The real issue, I believe, is that there is no accountability for almost anyone these days. If there is some upside in publishing or reporting on questionable science, and no downside, what is the incentive not to do it?
I agree. I think you captured the essence of the problem. There is no immediate cost to doing this (if people had an actual span of attention, most of the news sites would go banrkupt quickly), and clear benefits. There is no direct price to be paid for misleading people, and the accumulated costs are apparent only years later, in the form of growing distrust towards everything, which destroys society from inside.
Misinforming the public makes cool headlines, which bring in eyeballs, which bring in ad-dollars. I think it's as simple as that.
A neurosis is an unreasonable response to perceived reality which may indeed, as you say, not be what it appears. Solutions for neuroses must lie with the neurotic not with the stimuli held to be 'responsible' for them.
In 2013, a story made the front page of HN about high lead levels in imported rice (covered by some credible sources). For a few weeks, as a precaution, I avoided giving my children rice to eat (and I'm sure I wasn't the only one) and getting into a fairly heated argument with my spouse about it, who thought I was crazy. It later transpired that the entire story was based on miscalibrated equipment. Till today, there are news sources that haven't corrected the story (Google "lead in rice"). This was an extreme example, but I don't doubt that a lot of health stories suffer from real flaws, but are tougher to debunk.
Science is important, but science reporting (especially medical reporting) seems awful. No parent wants their child to be harmed, indeed, I've read comments on HN where people say they give their kids glass bottles because of the putative risks of phthalates. While I don't blame them for being careful, imagine what happens when the glass bottles shatter? It is very tempting for parents to panic and to "parent by headline", and it is obviously profitable for researchers and their institutions to generate maximal alarm. However, HN is my primary source of tech news, and I would prefer it if it wasn't clogged with possibly dubious health stories which perpetuate this cycle.
I don't think it's limited to science reporting. The more I learn, the easier is is to see the numerous errors made by journalists across all areas. Take Vox, it's an older link but they have been criticized for poor accuracy in reporting (and failing to prominently note corrections).[1]
The Royal Society has it right with their motto, nullius in verba: the news media, much like Wikipedia, is not a reliable source for facts.
[1]: http://theconcourse.deadspin.com/46-times-vox-totally-fucked...
Yes, and its corollary and my personal favorite...
I know I'm paranoid, but am I paranoid enough?
Here is an article about "How can we improve the accuracy with which evidence and statistics are reported"
http://runningres.com/blog/2015/3/18/how-can-we-improve-the-...
I guess you shouldn't believe any nutrition advice at all.
I see no reason not to believe in them.