Registered clinical trials make positive findings vanish
nature.com
nature.com
>>Prior to 2000, investigators had a greater opportunity to measure a range of variables and to select the most successful outcomes when reporting their results... Among the 25 preregistered trials published in 2000 or later, 12 reported significant, positive effects for cardiovascular-related variables other than the primary outcome.
That is, in most cases, there are large effects for some outcome, and if they get to choose the primary outcome after looking at some results, they could have been cherry-picking the outcome variables.
[1] http://journals.plos.org/plosone/article?id=10.1371/journal....
A lot of science is about serendipity.
Yes, that might happen. The more likely outcome though is that we are saved from a lot of drugs that don't work any better than chance (or even worse).
Not just could, would. Choosing your hypothesis after you run the experiment is (or at least should be) a cardinal sin in science for good reason. At the standard p-value cutoff of .05, even when there's absolutely no effect going on the probability of getting a spurious positive result when you do n comparisons is equal to 1 - (.95^n).
So that 5% chance of a type I error if you only look at one test statistic jumps to 40% if you look at ten, and to 72% if you look at 25.
Here's a nice piece of gonzo journalism that deals with this issue: http://io9.com/i-fooled-millions-into-thinking-chocolate-hel...
Choosing a hypothesis after the experiment is run is perfectly valid as long as your experiment is valid for that hypothesis. Besides, you would always run a new experiment again anyway.
Are you claiming that this statement is untrue, or irrelevant?
> Besides, you would always run a new experiment again anyway.
Then surely the initial experiment doesn't count. Otherwise, why are you re-running it?
Every experiment will support analysis through a set of hypotheses. Just because you didn't select all of those hypotheses before the experiment ran doesn't mean you can't select it after the experiment.
Imagine that an experiment has been run, but you do not know the results (or even what was done). Now you select a hypothesis, if the experiment required to validate the hypothesis is the same as what was run previously, you can now look at and use the results.
A hypothesis is like running a query against a database. Many queries are valid, even though the data may not have changed.
>Otherwise, why are you re-running it?
Science requires it. Doctrine from one-off experimentation is religion (hard to dump).
In exploratory research you collect a bunch of data, and then mine it for interesting associations that might merit further study. It's an essential part of the scientific process, but anything you find from doing it needs to be treated as extremely tentative because it's liable to produce spurious results at least as often as it finds genuine effects.
But that's not what this paper's talking about. It's talking about experiments that are being used to support the approval of new treatments and drugs. That's confirmatory research. In that realm you absolutely must paint the target on the wall before you throw your darts.
If the experiment is constructed correctly, it can support validation of multiple hypotheses.
I would completely believe that most experimentation being performed currently is not structured to make further exploitation possible.
Could you provide an example of an experimental structure that is immune to the statistics being discussed?
It should read:
I would completely believe that most experimentation being performed currently is not structured to make further exploitation, of the type desired/wished, possible.
There are almost certainly facts available that are not discovered/discussed from past experiments. Many of them are likely trivial and/or not what researchers would wish or hope that their data could tell them. However, they can still be mined.
In any case though, you would still re-run experiments to further validate/reject the hypotheses. That is simply basic science.
In other words, I run a study to determine if jellybeans cause acne[1]. The result is inconclusive (p < .05). Now--after the results are collected--I wish to check the correlation between color and acne. There are 20 colors. Would it be statistically sound to "correct" for my cherrypicking by setting p = 0.0025? That would result in
1 - (0.9975^20) ≈ 0.049
No. For one thing, how do you actually know n? If in your analysis you can make c independent binary choices, n would be on the order of 2^c. For most reasonable sequence of choices, n would be impractically large. And if you think you know n, how would you convince everyone else that your value of n is reasonable/trustworthy?
The xkcd case (which is solved by applying the Bonferroni correction) is special because there's only one test run in 20 experiments, so the correction is more straightforward.
There is a good paper about this garden of forking paths by Gelman and Loken [1].
[1] (pdf) http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
It's no panacea. When you use it you reduce your statistical power - in other words, your likelihood of detecting genuine effects goes down.
That's only the probability of getting a positive result by chance; the total probability of getting a positive result even if there is no effect is much higher.
C.f. http://blog.minitab.com/blog/adventures-in-statistics/how-to...
Right. And then you'd have to use proper statistical reasoning for that state of affairs. Which nobody ever does, cause they're not statisticians and it's complicated and it would reduce the chance of 'statistical significance'.
So they just use a standard calculation of statistical significance -- which is based on the assumption that you have picked a single hypothesis in advance and then done your test. So it's completely invalid to use it how everyone typically does.
Imagine you flip a coin 50 times. Then you see, okay, did I ever get 10 heads in a row? Nope? Okay, how about 5 heads followed by 5 tails? Nope. Okay.... try a couple dozen other things, oh, look, I got exactly 3 tails followed by exactly 3 heads followed by exactly 3 tails again! Let's run my test of statistical significance to see if that was just chance, or is likely significant -- oh hey, it's significant, this is likely a magic coin not random at all!
Nope. If you test everything you can think of, _something_ will come up as 'statistically significant', but it's not really, those tests of statistical significance -- which calculate how likely it is the results you got happened by random chance happenstance vs an actual correlation likely to be repeatable -- are no longer valid if you go hunting for significance like that.
Would this explain the propagation of, umm, data-challenged psychiatric drugs in the 80s and 90s?
I'm not sure what you're referring to. Could you unpack for those of us who haven't followed this but would like to learn?
A cursory google search brings up a decent review of the literature here (note, not peer reviewed to my knowledge but an article): http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4172306/
http://www.bmj.com/content/341/bmj.c4737
Pre-registration and mandatory publication would help prevent this.
If there is actual data to suggest who the drug works on best, and empirical tests to determine who that is, it can absolutely be tested within those limits and approved for use on that population.
What happens is the drug companies decide at stage III what patients to let into the trial. What they are trying to avoid is anything that will limit the market size once approved. This means they try to skate as close as they can to the effectiveness limit and still get the drug approved.
For example, if they have a new cancer drug that they know only works on cancers with certain mutations, what they will try to do is to test the drug in a broad population of patients including those without the mutation. Since for the approval process it does not really matter how effective the drug is (as long as overall it is statically significant) and that you can only market to the population you tested in, you will want to shove in as many “filler patients" as you think the drug can support. Due to the difficulty of designing large trials they often get the filler number wrong. I don’t condone this activity, but this is the commercial reality.
More fundamentally the pharmaceutical industry is being crushed between two very powerful forces - the cost of developing new drugs continues to get more expensive every year (mostly thanks to regulations), while the real market for each drug shrinks as sub-populations are identified for each disease. Combine this with a refusal of society and insurance companies to pay ever increasing drug prices and the outlook for the industry is not great.
If you want to get a good idea of what really goes on in the pharam industry have a read of the “In The Pipeline” blog [1]. Most of the real value is in the comments.
1. I honestly don't give drug companies that much integrity.
2. Sadly, I feel they systematically lied to the FDA in order to get their product to market? No, I don't think the average pharmaceutical research scientist wanted to poison people, but I have horrid feeling a lot of them went along with the sharade in order to make a living? I picture too many scientists thinking, "Well it doesn't work as we thought, but I'm really not hurting anyone because the Plecebo Effect was huge? So, in order to keep my job, I will not object to these results/statistics?". I have found that when it comes down to a person's livelihood, life--money; people don't raise their hands? They go into this weird, socially acceptable denial?("Hay it's legal!") That's why when even here on HN, I make sure to read the blurry statements.(I'm usually disappointed, but I want to hear the dissenters, and critics.)
3. I'll admit I threw the baby out with the bath water years ago, and I don't have hard evidence (in front of me collated, and ready to provide objective examples) to back up my intuition, and assertions, but I don't think one has to look too far for examples?
4. I grew up not questioning scientific integrity. I thought these institutions/people were the pillars of honesty!; what I have learned in the last decade, specifically about the Pharmacutical Industry, hurt me on so many levels--I literally feel like crying sometimes.
5. My understanding of statistics is fuzzy. To those that need a refresher, I found two books helpful: How to lie with Statistics, by Darrell Huff., and The Comic Book Guide to Statistics--don't know the author.
I myself don't anticipate that accounting for much though.
It's been noted that it's become much harder to find new drugs. Conventional explanations are that we're pretty good at this by now, and all the low hanging fruit has been picked (there's also fields like antibiotics where few are trying for unrelated issues).
More rigor in studies could be a cause, and/or there may be fewer successes because its become that much harder, and more iffy attempts are being made.
I think you mean, 'This casts doubt on the non-registered studies'.
Click click: http://amzn.com/0865478066
"Results for all cause mortality were similar. Prior to 2000, 24 trials reported all cause-mortality and 5 reported significant reductions in total mortality (25%), 18 were null (71%) and one (CAST) reported significant harm (Table 3). Following the year 2000, no study showed a significant benefit for total mortality."
So 24/30 pre-2000 trials they looked at reported all cause mortality. This is important because all cause mortality is the gold standard endpoint in a clinical trial. If you were cherry picking an outcome measure, this is the last one you would choose as it's the hardest one to get a positive result for.
This casts doubt on the paper's central thesis. How to explain the difference in pre and post registration all cause mortality? The possibilities are: 1. Trials performed after 2000 are genuinely less likely to be positive because of altered funding, lack of low hanging fruit etc. It is a well known problem that estimating the survival of the control group in a randomised trial based on historical data is usually an underestimate, which messes up the power calculation and the chance of a positive result. Beyond a certain point, it becomes infeasible to do a clinical trial to show a benefit over a very healthy control group, because you would need 100,000 patients. 2. After registration became mandatory, fraudulent investigators who make up their results stopped doing clinical trials.
The other point is that in most areas, there are only a few acceptable clinical trial endpoints. For example in cancer studies, there are really only 3: overall survival (time alive), event free survival (time until the cancer comes back or starts growing) and response rate (% of cases where the tumour shrinks by 20% or more). Cardiology is a bit different because there is less consensus about valid endpoints. Nevertheless, clinicians and regulatory bodies are pretty strict about which endpoints they consider meaningful.
So in my opinion the authors chose the subject area that would give them the most 'shocking' result (Cardiology, due to having more options for endpoints), glossed over the interpretation (why such a big difference in the gold standard endpoint that is supposed to be immune to manipulation?), and over-hyped the significance of the result.
And although I feel like a weird contrarian saying this, it's entirely possible that by selecting for treatments that are effective in other species, we may be inadvertently missing out on things that might work in humans.
Wait, what? Really?
If that's true, I can see only one explanation - with mice models, we can do far more aggressive research, and therefore we cover a bigger chunk of the parameter space. Some of the same research strategies may not be ethical with human models.
In regards looking for a cure for cancer we are like the drunk who lost their keys in the dark alleyway, but who is searching around the street lamp since the light is better. Until we actually start doing the right studies we will really struggle getting anywhere.
We never seem to realise while we are on the part of the curve that's accelerating (and things are great) that it's actually a bell curve. There are probably people writing things about this sort of stuff, how we can maybe start to overlap our understanding, maybe even measure what part of the curve we are on? Maybe get governments to invest money into new technologies just at the right points...
Not actually true. There are lots of examples of mines closing down because exploiting them further is not economical. Some of them restart operations when prices for the dug up commodity rise.
(A Google search for `mine reopening' easily finds a few examples in the news around the globe.)
If anyone know of a natural cancer cure study in mice or rats that has been published please get in contact with me (or post here) as I have been looking for one for years.