How to Make More Published Research True
plosmedicine.org
plosmedicine.org
Science today is full of bureaucratic nightmares - the publishing of a large amount of trivial results and the transformation of the most experienced scientists to uninvolved managers.
Until university positions and research grants stop being given out based on prior research results we won't be able to trust the research performed there.
There are millions of dollars on the line for researchers involved. It is the difference between a well paid career or a life of destitution while being a slave to huge student debt. It's no wonder they are blind to the flaws in their research.
I believe strongly that teaching positions and research grants should be given out based on criterions that are only incidental to research results.
Evaluate profs and grants based on:
1. domain knowledge (test the applicants) 2. math skills (test the applicants) 3. motivation and leadership 4. prior and current research proposals (but ignore results, especially the fact that they were published or not). 5. other skills such as written and oral communication
Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do well done research with interesting _questions_ regardless of the results.
If universities keep giving out jobs based on having generated interesting results in the past, they are going to keep getting researchers that ignore biases and publish whatever results are interesting whether they are true or not.
I come for a "search for physics beyond the standard model" background, where other than the neutrino mass (from the SNO collaboration, which I was part of) there hasn't been a positive result in decades. So there is already a good deal of focus on proposals rather than results, and yet almost all the issues I see in the biosciences (I jumped ship to genomics in the mid-00's) are also present in that area of physics.
Ergo, empirically, I'm doubtful that focusing on proposals rather than results will make much difference.
The difficulty is that science never makes economic sense for an individual. I spent a decade of my life measuring zero to higher and higher precision, and I know people who have spent entire careers doing so: putting new limits on branching ratios to exotic (which sounds so much better than "nonexistent") decays and so on. It was fun, although I took a year off in the middle to do some medical physics and imaging, which was even more fun because I actually got to measure phenomena that exist.
So when I read things about the paucity of "breakthrough discoveries" I think that mostly the low-hanging fruit have been picked and genomics turns out to be a whole lot harder and more of a slog than people expected, with a vast amount of uninteresting material to be waded through for the sake of a slow accumulation of knowledge that we are still a century away from putting to any very good use.
I don't know what an economically rational model for reward in such an environment is, and it's good that the article raises the issue and explores some alternative approaches, but I don't think there is any easy fix for the problem because I don't think science makes any economic sense. Just moral sense.
There is a very simple solution to the problem that completely eliminates gaming research results and publishing bias. Require that the statistical methodology is completely specified prior to any data acquisition. The paper is written before the data is acquired and it has some blank spots where the data will be filled in with a method that is completely mechanical (e.g. with a computer program that processes the data and spits out the figures that will be used to fill the blanks). Journals should decide whether to publish a paper or not based on the version without the data.
That is certainly an interesting proposal. How do you intend to assess competence in generating novel ideas (i.e. not testing for knowledge of existing work) if you ignore the candidate's track record?
What I think we need to do is reward negative results as much as positive results.
However, when you have a small pot of money, you do have to think about how do you make awards. On the one hand, you do ask the question, what are the chances that this would work and if it did, would it be "transformational"? That is, it may turn out to be a loss, but if it works, then it could really advance the field--in our committee, we did try to fund those kind of proposals over incremental advances.
Now, as to the question of reputation, I'm going to have to disagree. If you're going to give someone funds, how do you gamble? If they're young, then you can just look at their idea, their resources, and some indication that they have a chance at success. However, for a more senior researcher, they do have a track record. If they've received funds in the past and haven't accomplished anything with them, then why would you keep giving them money? If someone is publishing interesting results, then other people will try to duplicate and extend them If someone's research consistently fails these tests, then their reputation will suffer.
As for domain knowledge/math skills--I have to say that I think that this is relatively useless. Do you have any feeling for how many grant applications come in (along with multiple proposers)? And you want to test them across many subfields, etc.? These people managed to get their phDs, so if they are not competent, then that should have either showed up earlier, or in their publications.
I think there's a lot of merit in judging people by their results rather than simple tests which could be gamed.
Worse they are being given out almost exclusively based on where those past results are reported...
We definitely have a lot of room to improve though.
If you want to consider a problem in great depth before launching a startup to attack it, definitely go to grad school and do science for 4-5 years. If you think you have a good idea forget about grad school and just launch. (Obviously doesn't apply to things like bio where you need a lot of equipment, but as DIY science becomes more tractable this will go away.)
Especially interesting are the authors' ideas about the "reward system" in the current research world. If "value positions" like academic rank become irrelevant or even reduce reward, benefits would flow to research with greater scientific merit.
Effective science is much more a shared or collaborative effort than about proving who got there first. Too often authors stretch to make findings amount to an overarching explanation of the phenomena. Scientific quality would probably be improved just by scaling back conclusions and letting the data speak for itself.
The TL;DR is that one international user facility is creating a universal locator for data that you can cite in your publications to point people to the raw data, which is curated by the facility and openly accessible. This is extremely cool! It may be of limited use without metadata about how the experiment was performed, but has the potential to be rather useful.
The DOE in the US is moving to put more of the research published with its grant money into the public domain--either with preprints, or if the publishers can be encouraged, the final articles (this seems to be under development).
For some of my friends in statistics, there seems to be a move towards reproducible research where you try to preserve the toolchain that was used to reduce the data--this is considerably harder.
So, I think that there's progress being made, but it takes time for change. Also, unless there are requirements from funding institutions, it's hard for individuals to justify the time requirements involved. While most of my raw data is on the web and a I put out a number of my reduction tools, maintaining the whole tool chain would be rather difficult.
Henry Kissinger said the only thing you need to know about academia,
"Academic politics are so vicious precisely because the stakes are so small."
Care to elaborate? Because in my experience (non-bio, non-med), this is not true. It's certainly not perfect, but overall the peer-review system (again, based on my experience with it) results in scientific progress.
The goal would be to reward those who produce the most research with the highest impact. However, it often takes years or decades to determine which of the fashionable ideas circulating at any one time are the foundations upon which further knowledge will be built, and which will fade away.
A similar problem exists when discussing the mechanisms of rewarding Wall Street traders, bankers and politicians.
Since the impacts of the work performed today are so distant from realistic mechanisms of evaluation, proxies are instead used. In academia, this amounts to counting the number of papers a person publishes and the journals in which they are published.
So, are there alternative mechanisms that could be used? In what way could we harness the mechanisms of industry and capitalism to further "true science"?
A possibility, alluded to in the article linked, might be for researchers to makes "bets" on the outcome of their work. For example, they would lodge their predictions at some central place (perhaps using the mechanisms currently used for grant proposals), and if their predictions are correct, they get a reward. So, one would submit a grant application, some funding would be given, and if the results are correct, an extra tranche of money is delivered to the scientist (not their institute).
It would be necessary to ensure that they are not betting on results they already have (~insider trading).
That seems like a perverse incentive to me.
It wouldn't be sufficient for scientists to make predictions based on the work of others since the tools and so forth available to the candidate would not necessarily be widely accessible. However, once a scientist has 1) made a prediction, 2) demonstrated that their prediction holds out, and 3) that it is independently verified, then they should get rewarded at a rate above that of their peers who make predictions that don't hold out.
Edit: "prediction" in the above is easily replaced by "hypothesis".
I think that academia could make large steps forward by adopting some of the wonderful collaboration tools created by open source projects. Basic things like wikis, version control, accountability through open peer review, and open standards for data if required could really change the quality and the pace of innovation.
The current model was set up for an incredibly smaller community where everybody literally knew each other and the economics were different. What is needed now is more like a github for results/collaboration and python notebooks for analysis so others can reproduce and test.
Remember anyone can create a Facebook clone but it will never be Facebook. There is a very abstract currency in the human mind that no robot will ever solve.
I still think we're a long way off from automated dissection though. An AI performing these tasks would need very high level of situational awareness to be able to interpret the internal structure of a moving animal.