Reproducibility Crisis: The Plot Thickens | Hacker News Reader

Reproducibility Crisis: The Plot Thickens

blogs.discovermagazine.com

61 pointsDiabloD350 comments

From my perspective as a former academic there is no simple solution here. We have created a system where the pressure to publish first and ask questions later is totally overwhelming. This is a cancer within science that will ultimately destroy public support for funding scientific research.

If we are going to solve this problem we need to reduce the pressure on scientists to publish at all costs. This publish-or-die thinking is driven by the extreme competition for grants and tenure tracked positions. Given I think it unlikely that the amount of money available for research is going to quadruple anytime soon (this is about the amount of money we would need to put into the system to get rid of the publish-at-all-costs mentality) what can be done?

1. Break the link between publication and grant funding. We could move to a system where once you had proved that you could do research (say published a few papers) and could come up with a viable research idea that was doable assessed by a simple grant proposal, then you go into a pool and the grants allocated by lottery. We should aim to have 50% to 75% of all scientists in this pool. We then fund as many grants as we can given the money available.

2. Spend some of the current research money on long-term fellowships (say 10 years) that are awarded in the same lottery way as grants.

3. Drastically reduce PhD student training to get supply and demand back into line. e.g. make it such that a supervisor can not supervise more than one student at a time.

But let's not forget that a significant proportion of stakeholders have no interest in changing the system. The same "broken" system is actually a way to provide hidden subsidies for industry, decrease unemployment (in many countries), show the public that "research funding was increased by the government", and who knows what else.

Sure there is a strong vested interest in the status quo by those that have succeeded in it, but it has reached such an extreme level that it is killing science. My experience is that the successful scientists within the current system love science enough to change even if it puts them at a disadvantage.

Having said this we could grandfather in the current “winners” and move the funding to the new system over a long time period (say 20 years). We really need to do something as the current system will destroy science if we let it continue.

"the current system will destroy science"

I'm uncertain about this. The scientific method can luckily not be destroyed, and I wonder whether the self-destruction of academia is a good or bad thing. It is certainly not painless and will certainly take innocent victims.

If you consider non-equilibrium thermodynamics analogies, the quicker the system destroys itself the better it is, as it more or less _has to_ explode before it reorganises.

In Finland it has started already, as universities are required to get part of their funding from industry, and it's especially hard to accomplish in smaller cities.

The scientific can’t be destroyed, but the scientific infrastructure can be. To do experiments and make scientific breakthroughs you need access to a wide range of skills, equipment, and reagents. If this goes it will not be easy to get it back.

> We have created a system where the pressure to publish first and ask questions later is totally overwhelming.

Why can't people simply publish negative results?

> Break the link between publication and grant funding.

Same answer - in fact give extra priority to those who publish lots of negative results.

>> Why can't people simply publish negative results?

The journal's peer review process tends to reject them. Or so I'm told by a researcher friend of mine.

Because negative results are a dime a dozen. If you want to publish a negative result you have to do a lot more work to convince your peers that the thing you did was actually interesting. It's hard enough to do that with papers that show a real effect, scientists tend to be rather dismissive of other's work.

> Why can't people simply publish negative results?

Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work.

Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way.

Of course if you get a negative effect for an established "fact", as in this case it can be published. But you have to be really really sure otherwise the reviewers will just say you did your experiment wrong.

> Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work.

That seems to be the core of the problem. Reality doesn't have a bias toward results that are interesting, humans do.

> Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way.

So maybe they should.

> Why can't people simply publish negative results?

In addition to the other comments: because a negative result is usually much more likely than a negative one, it is very hard to be sure that the experimenter did not simply screw up the experiment.

Proving a negative rigorously is just as hard as proving a positive.

A lack of positive proof generally does not constitute a negative proof.

In principle there's no reason why they couldn't or shouln't. In practice, however, this leads to a non-trivial increase in the signal-to-noise ratio in academic journals.

I think it would be far better if major journals demanded datasets be made available and to make it easier to publish peer reviewed pieces that find anomalies or flatly contradict the conclusions of published papers.

Yes this would help, but it is not the driver of all these bad behaviours. We need to fix the root cause, not try and treat the symptoms.

The issue, as you probably correctly point out, is the pressure to publish too early and take shortcuts.

There is only so much money for scientific research as all areas of science are vast. Is it really surprising that there is so much pressure?

There has always been problems in science due to the demand being greater than the dollars available, the problem with the current system is we are rewarding people cutting corners and doing bad science. This needs to change.

Given limited funds I think we should not distribute them via a lottery. That just doesn't seem like the most efficient use of the money we have.

I fully agree with point 3 though. Currently we produce PhDs waay above replacement rate. Every tenured professor produces dozens of PhDs during his career, but his tenured position only becomes available once he retires. That's just not sustainable, no matter how much money you throw at universities and research institutes.

The reason why is to break the link between track record and grant funding. It has got so extreme that it has perverted the whole of science.

It just is not possible by peer review to determine the difference between a grant that is ranked in the top 10% and one that is ranked in the top 30%. Given we only have enough money to give less than 10% of all grants money, continuing the current process is just creating massive problems.

Not just a lottery, but it feels like randomness should be part of the process. The mechanisms and people we have in place to make decisions are horribly imperfect, so there's a lot of noise in the system as is. Many perfectly good grants don't get funded, often for unclear reasons. Adding randomness past a certain minimum bar just makes that noise explicit and ensures that it's not biased or spuriously correlated.

This would also deemphasize the importance of narrow skills specific to writing and getting grants. Right now, understanding the grant process and granting agency is probably just as important as, you know, coming up with good proposals. Not ideal!

Among other things, this could also help academics to spread out a bit—right now, research agendas are extremely fashion-driven, in large part because of how grants are awarded. Think about it like a search algorithm which, with some probability, randomly jumps out of its current state as a way to avoid getting stuck in local maxima.

Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few classes of qualified students each year, but only has room for one. At that point, it's distinctly non-obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have benefits. But this idea feels even more applicable to research grants.

Great post.

One thing about making the process (more) random is it would encourage people to take risks. When I was an academic I had to spend most of my time working on relatively low risk projects that I knew would generate results (and get me publications) or else I knew I would not get anymore grants. By removing this need I could have afforded to take risks that might have resulted in real breakthroughs. We need scientists to take risks and make the great discoveries.

> Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few sets of qualified students each year, and has to choose one. At that point, it's distinctly not obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have some benefits.

The only problem is that there's no sign that elite universities have any intention of choosing students on merit, or through any system which is blind to factors other than merit (plus pure chance). Harvard's selection system seems to be very good at its actual intended purpose: it's more like casting Glee than making a strict assessment of merit. But for the rare university which both has a large oversupply of very good applicants and actually wishes to operate admissions on merit then yes, randomness is the answer.

Another place that it would seem to make sense to apply randomness is in performance-related pay.

4. The grant agencies (e.g. the NSF) could use some of its money to pay some labs to reproduce the study during the end-of-grant review process, in the same way as it already pays reviewers and committees for the initial review process. This would add a direct pressure to publish things that are easy to reproduce.

Of course, that would mean even less grants...

5. Each PI should have a single grant at a time (and a long one), to end the "money competition" (that mirrors the publication competition) and spread the grants better.

In regards 4 the basic problem is it is really difficult to reproduce work. If an experiment does not work is it because the original paper was wrong, or is it because you are just not doing things correctly? The person best able to ensure everything is correct is the original investigator provided they have the time. This is what we need to fix.

If we were to move to a lottery process then there would be no need to limit anyone.

4. Require that papers report all possible correlations in order to make p-hacking impossible, to allow "stacking" the results of multiple studies and to make negative results more interesting.

5. Don't limit review to peer review, but involve people from different fields. What can be a well known source of bias in one field may be overlooked in another.

I don't entirely understand why the pressure to publish at all costs is the issue here. The article says:

> This particular plot is a statistical smoking gun, and suggests that the positive results from the original studies (black dots) were probably the result of p-hacking. They were chance findings, selectively published because they were positive.

From that, it seems to me like the problem isn't what the researchers published, it's what they didn't publish.

I'm a programmer, not a researcher, but in college chemistry we had marble notebooks that we wrote everything down in. The notebooks made it obvious if a page was torn out, everything had to be written in pen, and if we crossed something out, we had to cross it out in such a way that it was still legible. If we didn't include data in our results, we were expected to explain why. The most valuable thing I learned in those classes was the importance and practice of auditability.

I doubt that my classes were unique in this respect and I wonder if this might be the solution to the problem. I've long been a proponent of people simply publishing literally all their data now that internet hosting is so cheap, and only publishing syntheses of the data in journals, giving anyone the opportunity to verify the author's conclusions. Psychology is different due to the need for confidentiality for the people being studied, but the full data should be available to at least the peer reviewers before a study can be published.

EDIT: To add to this, elsewhere on this thread there's a lot of discussion about people not publishing negative results because they're not interesting. But reality doesn't have a bias towards results that are interesting, only humans do. I don't know how to solve this problem, but at least making publication of your non-interesting results a condition of publishing your interesting results would be a start.

Not publishing negative results is a symptom, not a cause, of the current extreme competition in science. Fix the competition issue and we will fix problems like the non-publication of negative results.

I'm not sure I see the causal relationship here. Why does increased competition mean that people are not publishing?

I think the problem is that the metrics which academics are competing to maximize are poor. If your metric is "How much interesting science has this person published?" you're going to get very different results from "How much accurate science has this person published?" Reducing competition doesn't change which question is being asked, it just reduces the importance of any metric at all. It seems likely to me that removing competition would give no incentive to publish anything measurable at all; after all, why publish if your career isn't connected to it?

Obviously some would publish for intrinsic reasons, and it's possible that stress levels due to competition are leading to shoddy work, but none of this amounts to an argument that less competition would lead to people publishing negative results.

Need to start incentivising negative results, reproducing results and even finding no results at all. It should be possible to publish to some kind of database a data analysis with methods and data even if you don't find anything, because then maybe other scientists could look at it and say "ok, this seem to not yield any result, but I could try to change it" adn he either finds it ant publishes, or doesn't find and publish anyway, so we start to get a collection of results and be much more skeptical when one of 8 studies, for example, finds effect of p barely over 0.005.

Yes. If journals are biased against negative results, then governments should move in. Why do we need "journals" with the implication of a limited page count per issue anyway; in this digital age a database or wikipedia-style format should be able to host any number of results.

And it should not mater that some governments won't fund suchan initiative: all is needed is one, or a single benefactor, or maybe a tv-show à la Mythbusters. ResearchBusters would be a hell of a show!!

I recommend reading responses to other branches of the grandparent that address this.

I agree that if people do decent studies and get null results, it would be nice if the data became publicly available so further studies can be seen in the context of previous negative results.

But given the need for novelty in results, I wonder how much work actually duplicates work previously undertaken with a null result. I suspect it's much less than many people in this discussion are assuming.

And secondly, there's the issue of forcing people to do decent write-ups. Study results are useless without a very detailed description of both the experimental setup and the data collection/analysis procedures that were followed. Those write-ups take time and effort, and for an unpublishable result many researchers won't find the motivation. One could argue that funding agencies and institutional IRBs must force researchers to write much of this ahead of running the experiment. That might help a bit, but will still leave substantial holes.

This is a direct result of "publish or perish," which basically gamifies academic advancement as a function of churning out publications. You get what you incentivize, and we wanted papers... we didn't say anything about correct reproducible results.

And reform will likewise be pursued as fast as congress reforms campaign finance. To many people have their livelihood invested in the castles they've built in the sky.

"their livelihood invested" - somebody said that "you can't be a true scientist if you don't rejoice when your model fails". As you suggest, a researcher today can't afford that. The best of science is done when somebody is "playing" curiously and without being invested in the expected results. In today's academia you just can't do that, or very rarely.

Man, that is a such a beautiful result. Ugly implications, but I love the analysis.

There should be another level of blinding: one team writes the study design, then sends the design off to a separate neutral group of practitioners who implement the study and report all results, regardless of outcome.

It'd be a much more expensive way to do science. But if cost matters and avoiding meaningless results doesn't, we shouldn't run studies, we should just flip a coin on every question and save the hassle of pretending.

I'd imagine if that had been done that Andrew Wakefield would never have gotten his MMR paper published.

That would likely be completely counterproductive. There's no such thing as a separate neutral group of practitioners. More likely what will happen is it will have a chilling effect on what science gets done. The Kinsey studies, for example, got as far as they did largely because they were under the radar for a long time. Imagine if, instead, Kinsey sent his study design off to a "separate neutral group of practitioners". They would likely have exposed the nature of the study and Kinsey would likely have been fired.

> They would likely have exposed the nature of the study and Kinsey would likely have been fired.

If you mean because the scientific community was reactionary and would have run him out, then that's a criticism of attitudes in the scientific community, not of the usefulness of extra layers of blind in studies.

Even so though, your characterization of the scientific community is a bit hyperbolic. Note that Kinsey gave many open discussions of his research area as early as 1935, and yet somehow retained his position in "swinging" Indiana.

> There's no such thing as a separate neutral group of practitioners.

I doubt that's knowable.

There are helpful degrees of neutrality, though, and you can eliminate certain specific sources of bias as they are identified. If there was no way to derive any objective results at all, then it wouldn't matter if we just flipped coins, because no studies would be worth anything.

Self-run studies are biased because significant results further a researcher's career. If you have some other team run studies without knowing the author, that team's bias is to establish its own credibility, which is orthogonal to just 'manufacture significant results.'

Could this also be explained by the "influence" experimenters had on their subjects? By influence I am supposing some kind of subtle cues... body language, or whatever. It also makes some sense that a smaller sample size (group of people) would be easier to influence.

I totally want proof that their paper is reproducible.

Psychology is not a science. Psychologists are not taught statistics in their curriculum. Why did anyone expect that they would come up with sound, statistically accurate results?

You can read any two psychology "textbooks" and get two differing opinions about how to address the same issues...

It's not a science folks. It's never been. It's always been an opinion-based practice, and this just proves the fraud being perpetrated on the people of this earth who trust these charlatans for help on matters ranging from personal mental health all the way to court "expert" testimony that can swing the decision of a judge or a jury one way or the other...

>Psychologists are not taught statistics in their curriculum.

That's just outright untrue. Maybe you have good points somewhere else in your comment but saying such nonsense makes me less inclined to hear you out.

This reply is to anyone reading this that takes the parent post seriously - the scientific method is the best tool we have for gaining knowledge, and it is effective especially with difficult subject matter such as psychology. Psychology isn't just science, it's difficult science.

BTW the only statistics I know are what I was forced to learn studying psych (although I was a computer science major, which required none)

How can you do computer science without any stats? Don't you study machine learning?

Many of us are kind of old. Machine learning wasn't much of a thing in the 90s, and I think has only become popular in the last few years.

Studying human behavior is just very difficult for any science (psychology is just at the fore). E.g. in programming languages, there are researchers that are trying to figure out how to empirically study human performance in relation to various programming features. It is quite difficult to get meaningful results, and experiments are expensive (you can't just put people in a box for later use!). No wonder most PL researchers focus on machine performance and theory...it is just easier that way.

It's an applied science, and they aren't charlatans.

It's not exclusively an applied science. Not in the least.

>Psychology is not a science.

False.

> Psychologists are not taught statistics in their curriculum.

False.

> You can read any two psychology "textbooks" and get two differing opinions about how to address the same issues...

Mind-bogglingly false; you have no idea what you're talking about.

I suspect you're either confusing psychology with psychoanalysis, or you're confusing experimental psychology with clinical psychology, as it's true that the DSM-IV had a serious disgnostic reliability problem (not sure what the sate of the DSM-V is).

In any case, understand that the term "psychology" encompases such research as this: http://cavlab.net/?lang=en

Non-social psychology (e.g. perception, attention, psychophysics, etc) has indeed become a pretty hard science.

Reply on news.ycombinator.com