Fixing Science - Systems and Politics

Some sort of validation mark that a study announced itself before being started would at least show me which ones I should ignore - because IMO this type of thing is ruining medicine, esp. pharma.

At this point, there's very little evidence that some major drugs, classes of drugs, and treatments work at all, and that's both scary and a systemic financial drain.

The author of the piece also identifies a massive, massive problem with science. The problem is essentially, tenure is hard to get, and many people want it. Tenure is dependent on highly cited papers in good journals. These journals have a bias toward significant results which pushes scientists to data-dredge looking for any significant result.

His system of pre-registration (which could be done on the ArXiV model, no government required) would allow us to have a much better idea of the things we're trying to study.

My field (psychology, though I suspect the problem is just as bad in other areas) has a massive problem in this area, as a counter-intuitive result gets published in a good journal, while high quality replications which show no significant effects either don't get published or get published in a much lower ranking journal.

This kind of registration becomes even more important when there are commercial interests riding on the outcome of a study, as in clinical trials. This (and the scandals) is presumably why such a system exists for clinical trials.

Finally, I don't see how registering studies and their designs before running the tests creates a centralised approach to science. The research is all planned beforehand (for ethics committees at the very least), so there's no extra work involved. It also increases our trust in results, as the ones which make it through were predicted in advance, while the weird findings can be replicated.

On a personal level, I can see the pressures to hypothesise after the results are known (HARK) and with tenure decisions looming, I can understand why people do it. Its horribly wrong, so this system would at least cut down on that behaviour.

Full disclosure: almost every time I carry out research, the results are opposite to what I expect, so I am a somewhat biased participant in this debate.

If science could be done better, yes, we should fix it. The author identifies a very real problem - publication bias and data mining, and proposes a solution, namely pre-registration of experiments.

Only one of his mechanisms for enacting the solution (#1, "clinicaltrials.gov option") is centralized.

There are many different fields in science, each of which needs its own rules / conventions. For example, physics is a field with a really long history, where the scientific method has been perfected over the centuries by a number of very smart people doing thousands of experiments. Still, there are errors that take some time to be resolved (the recent neutrino blunder - although it should also be said that it was resolved precisely because physics is such a well-organized field of science).

Other sciences, e.g. medicine, are much younger (the scientific method being formalized only in the last century), mistakes are more costly (people might die), experiments take longer and the effects of new discoveries are not visible immediately (could take a whole generation). E.g. I still wonder how will the prevalence of mobile phones affect the fertility of the people who have been carrying a mobile phone in their pocket (next to ovaries/testicles) for their whole lives, but there is no answer and there won't be one for at least another 20 years.

Only the future generations will be able to assess whether our science is broken or not, and whether we're doing anything to fix it.

This is a real problem that has been getting a lot of attention lately: http://www.asm.org/index.php/news-room/release032712b.html