Hacker News Reader

But let's not forget that a significant proportion of stakeholders have no interest in changing the system. The same "broken" system is actually a way to provide hidden subsidies for industry, decrease unemployment (in many countries), show the public that "research funding was increased by the government", and who knows what else.

Sure there is a strong vested interest in the status quo by those that have succeeded in it, but it has reached such an extreme level that it is killing science. My experience is that the successful scientists within the current system love science enough to change even if it puts them at a disadvantage.

Having said this we could grandfather in the current “winners” and move the funding to the new system over a long time period (say 20 years). We really need to do something as the current system will destroy science if we let it continue.

"the current system will destroy science"

I'm uncertain about this. The scientific method can luckily not be destroyed, and I wonder whether the self-destruction of academia is a good or bad thing. It is certainly not painless and will certainly take innocent victims.

If you consider non-equilibrium thermodynamics analogies, the quicker the system destroys itself the better it is, as it more or less _has to_ explode before it reorganises.

In Finland it has started already, as universities are required to get part of their funding from industry, and it's especially hard to accomplish in smaller cities.

The scientific can’t be destroyed, but the scientific infrastructure can be. To do experiments and make scientific breakthroughs you need access to a wide range of skills, equipment, and reagents. If this goes it will not be easy to get it back.

Given limited funds I think we should not distribute them via a lottery. That just doesn't seem like the most efficient use of the money we have.

I fully agree with point 3 though. Currently we produce PhDs waay above replacement rate. Every tenured professor produces dozens of PhDs during his career, but his tenured position only becomes available once he retires. That's just not sustainable, no matter how much money you throw at universities and research institutes.

The reason why is to break the link between track record and grant funding. It has got so extreme that it has perverted the whole of science.

It just is not possible by peer review to determine the difference between a grant that is ranked in the top 10% and one that is ranked in the top 30%. Given we only have enough money to give less than 10% of all grants money, continuing the current process is just creating massive problems.

Not just a lottery, but it feels like randomness should be part of the process. The mechanisms and people we have in place to make decisions are horribly imperfect, so there's a lot of noise in the system as is. Many perfectly good grants don't get funded, often for unclear reasons. Adding randomness past a certain minimum bar just makes that noise explicit and ensures that it's not biased or spuriously correlated.

This would also deemphasize the importance of narrow skills specific to writing and getting grants. Right now, understanding the grant process and granting agency is probably just as important as, you know, coming up with good proposals. Not ideal!

Among other things, this could also help academics to spread out a bit—right now, research agendas are extremely fashion-driven, in large part because of how grants are awarded. Think about it like a search algorithm which, with some probability, randomly jumps out of its current state as a way to avoid getting stuck in local maxima.

Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few classes of qualified students each year, but only has room for one. At that point, it's distinctly non-obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have benefits. But this idea feels even more applicable to research grants.

Great post.

One thing about making the process (more) random is it would encourage people to take risks. When I was an academic I had to spend most of my time working on relatively low risk projects that I knew would generate results (and get me publications) or else I knew I would not get anymore grants. By removing this need I could have afforded to take risks that might have resulted in real breakthroughs. We need scientists to take risks and make the great discoveries.

> Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few sets of qualified students each year, and has to choose one. At that point, it's distinctly not obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have some benefits.

The only problem is that there's no sign that elite universities have any intention of choosing students on merit, or through any system which is blind to factors other than merit (plus pure chance). Harvard's selection system seems to be very good at its actual intended purpose: it's more like casting Glee than making a strict assessment of merit. But for the rare university which both has a large oversupply of very good applicants and actually wishes to operate admissions on merit then yes, randomness is the answer.

Another place that it would seem to make sense to apply randomness is in performance-related pay.

4. The grant agencies (e.g. the NSF) could use some of its money to pay some labs to reproduce the study during the end-of-grant review process, in the same way as it already pays reviewers and committees for the initial review process. This would add a direct pressure to publish things that are easy to reproduce.

Of course, that would mean even less grants...

5. Each PI should have a single grant at a time (and a long one), to end the "money competition" (that mirrors the publication competition) and spread the grants better.

In regards 4 the basic problem is it is really difficult to reproduce work. If an experiment does not work is it because the original paper was wrong, or is it because you are just not doing things correctly? The person best able to ensure everything is correct is the original investigator provided they have the time. This is what we need to fix.

If we were to move to a lottery process then there would be no need to limit anyone.

I don't entirely understand why the pressure to publish at all costs is the issue here. The article says:

> This particular plot is a statistical smoking gun, and suggests that the positive results from the original studies (black dots) were probably the result of p-hacking. They were chance findings, selectively published because they were positive.

From that, it seems to me like the problem isn't what the researchers published, it's what they didn't publish.

I'm a programmer, not a researcher, but in college chemistry we had marble notebooks that we wrote everything down in. The notebooks made it obvious if a page was torn out, everything had to be written in pen, and if we crossed something out, we had to cross it out in such a way that it was still legible. If we didn't include data in our results, we were expected to explain why. The most valuable thing I learned in those classes was the importance and practice of auditability.

I doubt that my classes were unique in this respect and I wonder if this might be the solution to the problem. I've long been a proponent of people simply publishing literally all their data now that internet hosting is so cheap, and only publishing syntheses of the data in journals, giving anyone the opportunity to verify the author's conclusions. Psychology is different due to the need for confidentiality for the people being studied, but the full data should be available to at least the peer reviewers before a study can be published.

EDIT: To add to this, elsewhere on this thread there's a lot of discussion about people not publishing negative results because they're not interesting. But reality doesn't have a bias towards results that are interesting, only humans do. I don't know how to solve this problem, but at least making publication of your non-interesting results a condition of publishing your interesting results would be a start.

Not publishing negative results is a symptom, not a cause, of the current extreme competition in science. Fix the competition issue and we will fix problems like the non-publication of negative results.

I'm not sure I see the causal relationship here. Why does increased competition mean that people are not publishing?

I think the problem is that the metrics which academics are competing to maximize are poor. If your metric is "How much interesting science has this person published?" you're going to get very different results from "How much accurate science has this person published?" Reducing competition doesn't change which question is being asked, it just reduces the importance of any metric at all. It seems likely to me that removing competition would give no incentive to publish anything measurable at all; after all, why publish if your career isn't connected to it?

Obviously some would publish for intrinsic reasons, and it's possible that stress levels due to competition are leading to shoddy work, but none of this amounts to an argument that less competition would lead to people publishing negative results.

> We have created a system where the pressure to publish first and ask questions later is totally overwhelming.

Why can't people simply publish negative results?

> Break the link between publication and grant funding.

Same answer - in fact give extra priority to those who publish lots of negative results.

>> Why can't people simply publish negative results?

The journal's peer review process tends to reject them. Or so I'm told by a researcher friend of mine.

Because negative results are a dime a dozen. If you want to publish a negative result you have to do a lot more work to convince your peers that the thing you did was actually interesting. It's hard enough to do that with papers that show a real effect, scientists tend to be rather dismissive of other's work.

> Why can't people simply publish negative results?

Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work.

Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way.

Of course if you get a negative effect for an established "fact", as in this case it can be published. But you have to be really really sure otherwise the reviewers will just say you did your experiment wrong.

> Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work.

That seems to be the core of the problem. Reality doesn't have a bias toward results that are interesting, humans do.

> Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way.

So maybe they should.

> Why can't people simply publish negative results?

In addition to the other comments: because a negative result is usually much more likely than a negative one, it is very hard to be sure that the experimenter did not simply screw up the experiment.

Proving a negative rigorously is just as hard as proving a positive.

A lack of positive proof generally does not constitute a negative proof.

In principle there's no reason why they couldn't or shouln't. In practice, however, this leads to a non-trivial increase in the signal-to-noise ratio in academic journals.

I think it would be far better if major journals demanded datasets be made available and to make it easier to publish peer reviewed pieces that find anomalies or flatly contradict the conclusions of published papers.

Yes this would help, but it is not the driver of all these bad behaviours. We need to fix the root cause, not try and treat the symptoms.

The issue, as you probably correctly point out, is the pressure to publish too early and take shortcuts.

There is only so much money for scientific research as all areas of science are vast. Is it really surprising that there is so much pressure?

There has always been problems in science due to the demand being greater than the dollars available, the problem with the current system is we are rewarding people cutting corners and doing bad science. This needs to change.

4. Require that papers report all possible correlations in order to make p-hacking impossible, to allow "stacking" the results of multiple studies and to make negative results more interesting.

5. Don't limit review to peer review, but involve people from different fields. What can be a well known source of bias in one field may be overlooked in another.