Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
blog.scienceexchange.com
blog.scienceexchange.com
The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science.
Next on the list, open source repositories for protocols of experiments! Maybe someone surprises me with a link to an existing solution :).
Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is still rising throughout the developed countries of the world,[1] so at least the trend lines look good for continued improvements in health as medical research is incrementally improved. Girls born since 2000 in the developed world are more likely than not to reach the age of 100, with boys likely to enjoy lifespans almost as long. The article "The Biodemography of Human Ageing"[2] by James Vaupel, originally published in the journal Nature in 2010, is a good current reference on the subject. Vaupel is one of the leading scholars on the demography of aging and how to adjust for time trends in life expectancy. His striking finding is "Humans are living longer than ever before. In fact, newborn children in high-income countries can expect to live to more than 100 years. Starting in the mid-1800s, human longevity has increased dramatically and life expectancy is increasing by an average of six hours a day."[3] An article in a series on Slate, "Why Are You Not Dead Yet? Life expectancy doubled in past 150 years. Here’s why"[4] makes clear that it's progress here, and progress there, and a lot of little things adding up that have resulted in the dramatic increases in healthy lifespan over the last few generations.
[1] http://www.scientificamerican.com/article.cfm?id=longevity-w...
[2] http://www.demographic-challenge.com/files/downloads/2eb51e2...
[3] http://www.prb.org/Journalists/Webcasts/2010/humanlongevity....
[4] http://www.slate.com/articles/health_and_science/science_of_...
There's an entire field of researchers researching researchers, so it's trivially easy to put together a list of common research errors and their prevalence. The only difficult thing is calculating the degree of independence. But there isn't any evidence, so far as I know, that these errors are highly correlated. Nor is there much reason to beleive that they are, since presumably the more errors a study has, the less likely it is to be published. Also, we already know from Ioannnidis et al. that the percentage of medical studies that can't be replicated is (iirc) in the 80-90% range. And just because a study can be replicated doesn't mean it's accurate, so 10-20% is really an upper bound on accuracy.
> Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is still rising throughout the developed countries of the world
But as I've posted previously, and as you can find trivially easily via Google, of the 30 year increase in life expectancy in the US, a minimum of 25 of those years have come from public health improvements rather than from modern medicine.
Do you categorize vaccination as modern medicine, or as public health? Is there no overlap, some overlap, or much overlap in the concepts of "modern medicine" and "public health"?
It depends on what point you're trying to make. E.g. if you're talking about contribution of pharmaceutical drugs to longevity, then it doesn't make sense to include vaccines since they're not drugs. If you're talking about the contributions of modern medicine as a whole, then it make sense to include all the vaccines developed roughly since Dr. Gold popularized the gold standard clinical trials, so sometime in the mid to late 1950s. Or if you're talking about the contributions of the western medical system as a whole since its inception, then it probably makes sense to include all vaccines, etc.
Assuming you are getting the 25 years from the CDC[1], you are being obtuse. Immunizations is the first thing listed. Vaccines are a component of modern medicine. Control of Infectious diseases includes antibiotics, which are medicine. Clicking on "Declines in Deaths from Heart Disease and Stroke" shows that it includes "an increase in the percentage of persons with hypertension who have the condition treated and controlled" using pharmaceuticals, definitively modern medicine.
In summary, these 25 years aren't coming from just getting people to quit smoking. There's a large component that is modern medicine.
Given that the argument was about the reliability of modern medical studies, it would probably be only reasonable to include studies that came onto the market after the era of plabeco controlled RCTs. But many if not most of the most impactful vaccines came out before then. Of the remaining important vaccines, they account for only a tiny fraction of the literature.
But more importantly, it's not even a valid argument to begin with since the increase in life expectancy has nothing to do with the percentage of medical papers that are flawed.
I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.
Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem.
Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.
We felt the preclinical stage was more important to validate, as much of the research is based on academic studies with over 60% failure rate for replication (see http://www.nature.com/nature/journal/v483/n7391/full/483531a...)
By clinical trials, a lot of the pharma companies have already done initial validation, so extra validation is redundant. We hope a reproducibility system to validate preclinical research can help patient groups, foundations, and industry better identify reproducible oncology targets, and hence the focus on preclinical studies (which in turn are less expensive).
You are right that the per study costs are cheaper than normal as the original protocol is already developed, and there is less trial and error involved. In certain cases, we may not need to validate every aspect of a study as well, only those specific experiments that warrant replication. And depending on the study, those experiments may vary in cost (i.e. a mouse model component would be expensive, but RNA extraction and western blot testing relatively inexpensive).
Additionally, we will leverage expert labs on Science Exchange which perform experiments regularly on a contract basis, allowing for bulk savings on reagents and supplies.
Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124
and privately: the matter of fact is, pharma companies are unhappy, because according to them, over 70% is irreproducible (euphemism for BS).
This is why ie. BMJ will only accept papers from January on that publish evidence ie. data also. One wonders what happened before...
BMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f5975
And where is the peer review in this process? I suppose as soon as something turns up unreproducible we will find out.
But you seem to be positing the reverse: a hypothetical case where we assume that the results of the original studies were in fact correct (i.e. ground truth), and we want a sort of false-negative rate, the chance that a fact would not be confirmed by a replication study, despite actually being true. That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power
The overall failure-to-replicate rate would involve both aspects, with four possible outcomes at varying likelihood: 1) a correct result that was confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect result that was confirmed anyway; and 4) an incorrect result that failed to be confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2 and #3.
Not quite. Assume P of the tests are truly positive, and N are truly negative.
We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05 x N / (0.05 x N + A x P).
It's not just power, you also need prior odds if you want to make an unconditional estimate. This is pretty much the core of Ioannides's famous paper http://www.plosmedicine.org/article/info:doi/10.1371/journal... - even after taking power into account, the replication rate could be anywhere from 0 to say 80% depending on the prior odds.
I think that in general you're right, with the assumption that p=0.05. However, each of the studies being replicated is composed of many intricate experiments. We will be reproducing most of those experiments, so while the odds of a single validation experiment failing are certainly not zero, I think the chances of an entire paper's worth of reproduction experiments being false negatives is much less likely. All of the protocols and reagents used, and the data from the replication studies, will be publicly available when they are complete, so that everyone can see what we did and how we did it.
Oh, dear. This is very much not a good sign, since answer to grandparent's question is "no" as pointed out above and anyone who knows statistics should see that immediately. (Checks Science Exchange site.) Okay, Bilal Mahmood is "customer support and outreach" so this is not necessarily fatal. Still worrisome.