> It's not that the results are completely invalid
Especially if only 11% of them are reproducible, they're only 89% invalid.
I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.
Especially if only 11% of them are reproducible, they're only 89% invalid.
I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.
We could just give up, or we could try to continue study of something incredibly complex. Given that we are gaining some ground, it's clearly not a useless endeavor, it's just an extremely difficult one.
What was being tested was a different lab, with different materials, trying to get the "same" results, for some definition of same. If you give 100 programmers an algorithms book, and tell them to produce code for a binary search, and only 25% of the programmers are able to make something that works, does that mean that binary search is only 25% reproducible?
If five different companies benchmark five different web frameworks for their application, and come to 2-4 different answers about which one is the 'best,' does that mean that the benchmarks are not reproducible? Of course not.
What's being highlighted here in this study is the extreme diversity of biological models. And one doesn't necessarily expect exact reproducibility in other people's hands, because we simply don't have technology to characterize every single aspect of a biological model, and it's impossible sometimes to even recreate the exact same biological context. Is something "reproducible" if it means that it replicates in 5% of other cell lines, 25% of other cell lines?
I don't follow you here. The above does not seem to be the current meaning of "reproducible":
http://en.wikipedia.org/wiki/Reproducibility
The same person doing the same experiment is repeatable, not reproducible. And I don't believe even the repeatable bar has been met, as very few projects have funding to do the same experiment twice.
The fact that a given investigator can "repeat" his experiment have very low weight among professional scientists, because we are all human. Irving Langmuir's famous talk about Pathological Science, and especially the sad story of N-rays, is a warning to every scientist.
Part of this is that science isn't about experimentation, it's about observation. We perform experiments when possible so that we have more stuff to observe, or more controlled events.
Much of astronomy, atmospheric physics, geology, medicine, the "soft" sciences and I'm sure plenty of other "hard" fields are at the mercy of certain phenomena having sweet FA for data points. And I'm sure they all do what astrophysicists do: make sure that what we do have plenty of data for works; make our extrapolations with as few assumptions/rounding errors as we can; and revisit existing models anytime we find a new data point.
It's as scientific as anything it's simply going to take longer to sort out in some cases.
Apparently people take that crap seriously.
Cosmology on the other hand...
Reproducibility isn't about calling out people whose work isn't reproducible, it's about identifying and promoting the most robust stuff.
The current test (if I remember my philosophy of science correctly is about falsifiability - it's not science if its claims can't be disproven. From this perspective, bad experiments are still science - someone predicted that similar experiments would behave similarly, and their prediction was falsified. This is how science is supposed to work.
It gets problematic when any failure to reproduce instinctively gets explained away as experimental error on the part of the second experimenter. Even worse is when experimenters (as in this case) work to have failures to reproduce hidden from the scientific community (the authors of this study had to sign contracts that they would not identify specific failing studies before they were given the necessary data about experimental procedure.
The problem is people are not trying to reproduce results which harms the field and slows everything down.
These Amgen researcher had to sign agreements that they would not publish the results of their attempts to reproduce these experiments before they could get enough detail to attempt to reproduce them. Clearly this is not how science is supposed to work but it is exactly how businesses operate. Very sad.
This is a problem that is getting worse as funding is getting cut more, people feel they have a need to get a paper out, regardless of the results. You get positive results, make a story up about it, then run to publish it before trying to reproduce it or look further into the data. While this doesn't happen in every lab, I'm unhappy to say that I've seen this happen in many "high impact" labs.
I don't think the problem is lack of funding but screwed up incentives. When medical reaseach became focused on funding the quality of the results suffered. And if the vast majority of landmark cancer research can't be reproduced much of that money was wasted.
The solution will require a huge cultural change which may be impossible. However step one is recognizing the problem. And some efforts are already underway such as journals like PLoS which publish negative results and more recently The Reproducibility Project and Reproducibility Initiative [2,3]. Still it will be difficult.
[1] http://officeofbudget.od.nih.gov/pdfs/spending_history/Mecha...
[2] http://www.openscienceframework.org/project/EZcUj/wiki/home
Already funded grants are getting cut ~20% across the board. There is a ton of cuts going on right now to the NIH budget, google it and take a look. This has been happening for years now, trying to get an RO1 (large research grant) is becoming more and more difficult and it isn't helped by the constant changing in requirements.
Every other point I agree 100% with, the culture change has to happen. Nothing is impossible, and it just takes the right people to make the right things to happen. Everyone recognizes the problem, I can't tell you how many times I hear people complaining about the same problems over and over. The problem is that they aren't taking action, and with no action, nothing is going to take place. While there are many efforts in place (my project being one of them, http://omnisci.org), they need to be implemented properly. The same rule of startups applies to science. The ideas/concept means nothing with proper execution.
While the culture change may be slow, the academic world is having a really hard time keeping up. NIH is also fighting to stay afloat. I have a few friends who work as program officers and they really have a negative outlook on the future of research funding.
Good luck with omnisci.org this is the sort of thing that would help: open sharing of data, techniques and negative results. If this was the norm things could be very different. But one thing I have learned is it is very hard to change and organizations culture.