The Need for Reproducibility in Academic Research
blog.scienceexchange.com
blog.scienceexchange.com
One aspect of science that doesn't get much attention in this debate is the role of the scientist as an ethical and idealistic actor; to be a scientist is (or was) have a higher calling, to help humanity get closer to the truth. And this is crucial to science itself because scientists need to be able to trust other scientists. And neither Everyone-watches-everyone-style trust nor you-will-be-punished-harshly-if-caught trust works. You need I-do-it-because-I-believe-in-it trust to make science work.
Now, the more that graduate students are made disposable, the more that professors live in a ruthless, sink-or-swim environment and so-forth, the less a scientist is likely to remain an idealist interested first and foremost in discovering the truth and the less that crucial element of trust will remain.
The latest fad is "outsourcing science". If we want to make science less broken, it seems like we should be going in the opposite direction.
Your policy of "I do it because I believe in it" seems quite unstable - under such a policy, it seems bad actors can come in and take advantage of things with no mechanism to prevent it.
I'm not necessarily against the "make life nicer for scientists" policy you seem to be advocating, I'm just not understanding your reasoning.
Technically, science can work if your colleagues are untrustworthy. This is one of its big, famous features. Over the centuries, scientists have published a great many howlers, ranging from honest mistakes to rushed procedures, deliberate disinformation, and straight-up fraudulent data. These things get caught, their perpetrators get punished to some extent, and science makes progress. Eventually.
The problem is that "eventually" can be a really long time: Years or even decades. (The Piltdown Man hoax wasn't exposed for forty years.) In the meantime, bad science will confuse the analysis, corrupt the textbooks, and injure the careers of a few unlucky grad students. It will waste a great deal of time and money, perhaps that of the most prominent people in the field.
For example, when cold fusion hit in 1989 dozens of scientists dropped everything for at least six months to try and replicate it. Millions of dollars were spent. Obviously, while those folks were tinkering with cold fusion they weren't tinkering with anything more interesting or useful.
We've made a lot of progress since Galileo, the frontier has moved a long way, so it takes more than a couple of pendulums to replicate most modern scientific papers. It could take half a decade, the entire productive career of one or two grad students, an entire research grant, a lab full of equipment, and the lives of two hundred mice just to replicate one paper. So the mutual trust is essential for speed: You have to be able to gamble your time on the results of other people's experiments with some hope of a positive return [1], or the speed of science slows down to the speed of one person's work. (Even that could work - you can discover things even as a sole practitioner - but it would be incredibly slow. Particularly because a scientist working without good criticism will make mistakes, lots of them.)
---
[1] The return will never be 100%. One of the things that disturbed me as a physicist switching to biology was that even the best biology papers are inevitably riddled with likely sources of error: The subject is just too complex to control everything perfectly. There are, for example, systematic sources of error that underlie entire fields, like the fact that most results are tested either on one highly inbred species of lab animal or on lines of human cells that have been selected to thrive well in dishes, and which are therefore, at some level, unlike any cells seen in any living human. So, science is inevitably a kind of gambling: Will you see consistent and useful results from this particular corner of experiment space? If the thing kills cancer in the dish, will it work in mice? if it works in mice, will it work in humans? if it works for 10% of humans, will it work for 40%? You gamble and you hope. You hope that you aren't wasting your grant, your career, or your entire field. The good news is that we do tend to win, in the long run, but anything that improves the odds on the bet is helpful.
Maybe I've been too close to science for too long but the whole line of argument seems so obvious to me that my reaction is "I don't know where to begin" when someone implies science could be done through coercion.
Of course, just because it makes no sense and is terrifyingly sociopathic doesn't mean it doesn't happen. Among other things: Mental illness happens. It's scary when it does.
The issue, I suspect, stems from the nature of publishing: top-tier journals only publish "interesting" research, which means reproducing research is less welcomed and if performed needs to be accompanied by a serious value-add.
There is no incentive to reproduce. It makes it more difficult to publish. It doesn't lead to tenure. Why bother?
I was once implementing an algorithm I've read about in a paper and could not reproduce the published results. After lots of frustration I contacted the author just to find out that the associated dataset he published on his website was modified and is no longer suitable for testing of the original research. He also said he has lost the original dataset, which meant I could neither verify the correctness of my implementation, nor his published results...
Not doing so seems equivalent to publishing a paper about applications and characteristics of a new mathematical proof without actually including the proof.
Not too long ago I emailed someone at UIUC about a tool they wrote which was mentioned in a research paper I ran into online. I wanted to see if their method really was much better than previous ones, and if the trade-offs they made were worth it.
Did I get it? No. Instead, they sent me a link to some new company founded based on the tool. I apparently had to be a "researcher" to receive the magical tarball.
It also seemed to me to be a conflict of interest for a Professor to be working for a University and company at the same time - all while selectively choosing who can and cannot get access to their results.
Such a response from a researcher should be extremely disconcerting.
It would seem a good fit for younger scientists at less prestigious (poorer) institutions, looking to get some form of objectively validated reputation they can cite for career reasons while they work to catch up to some part of the advancing-edge of research in their field.
With enough traffic of scientists openly trying to reproduce the results in other journals (sometimes succeeding, sometimes not), it would be only a matter of time before some journals that have a bias toward "surprising" to the detriment of "true" get bitten by repeated results in the TRR that go against what they published. So long term, that would be a method by which TRR could garner some respect in fields.
In another example a very promising cancer drug prediction algorithm (with fascinating in vitro results tested by an affiliated lab) was abandoned because of a key researcher's untimely death and the complete lack of version control anywhere in the lab. The paper had already been published (thankfully) but we literally had no idea where the code and the intermediary data were. We had a ~5,000 node GPFS cluster with rolling backups but it didn't help at all because all the development was done locally; the situation was the same across the lab. The decision of the PI in the wake of this compound tragedy was to have lab members pair up and "cross train" each other for an hour and verbally tell them where they kept their important data.
Referring to the corrupted data issue I personally experienced, I unfortunately discovered it the night before a multi-departmental research presentation. There were numerous reversed edges in a large digraph due to improper integration of two data sets before my involvement (I was also at fault for trusting internal data). I told the PI about it in the morning since the problem was so deep and said I couldn't present anything because every single result of the past year was invalidated by the bug I had found. His response: present anyway. I refused. That did not go over well.
I'd like to see every computational paper (especially in biology where these methods end up influencing human clinical medicine) include all source code in a public repository but it isn't going to happen. Labs would lose their edge if they had to tell competitors what model weights they had iterated to in creating their newest prediction algorithms and university technology transfer departments would have greater difficulties patenting these methods and selling them to drug companies. The current model will not change but a new one might supplant it.
I wasn't on the cancer drug prediction project but I probably know enough about it to reconstruct it. It actually seems like a great candidate for an open source project.
http://nlpers.blogspot.com/2011/03/some-thoughts-on-suppleme...