Comment by meow1032 | Hacker News Reader

Comment by meow1032 | Hacker News Reader

Parent Full thread

> What might work is that we require replication work for a PhD

I don't think this will work. All it will do is devalue the value of replication studies because only PHD students do replication studies. It's also not in their best interest especially if they dispute findings of established researchers.

Also, we have to get away from the idea that the scientist's job is to think and write, and literally all of the other work can be shuffled off onto low wage (or no wage), low status workers. This is one of the biggest reasons that science is going through such a crisis. If you want enough papers to consistently get grants you probably need at least 4/5 PHD students every few years. This causes a massive glut in the job market. It also dissociates scientists from their work. I've met esteemed computational biologists who could barely work a computer. All of their code was written, run, and analyzed by graduate students or post docs. They were competent enough at statistics, but that level of abstraction from the actual work is troubling.

Requiring replication work for a PhD seems like a great idea. PhD programs already use a mandatory exercise—the qualifying exam—to check a student's competence, with ambiguous effectiveness. Turning the qualifying exam into a replication study seems like a win: it tests the student's ability to do their actual job rather than pass an abstract test, and produces output that is useful both to the student and the community. The qualifying exam committee (usually ~ 4 PIs from different labs) can do quality control on the replication.

> All it will do is devalue the value of replication studies because only PHD students do replication studies. It's also not in their best interest especially if they dispute findings of established researchers.

Most studies are done by students regardless, so it seems unlikely that replication studies would be devalued merely because they're done by students. Although disputing the findings of established researchers can be risky, they would be publishing jointly with their PI (or, with the above implementation, multiple PIs), not alone with no support. Few students want to stay in academia, so it usually doesn't matter to them if a professor at some other institution gets offended. Most importantly, if everyone is doing replication studies, there will be so many disputations flying around that any particular person is less likely to be singled out for retaliation.

It sounds like what you're suggesting would be functionally equivalent to PI-led replications, which I would agree is a good idea. There are still some practical problems though.

1. Studies can be much more expensive than most people think. In my field, a moderately sized study can easily cost $100,000+ if you're only accounting for up front cost (e.g. use of equipment, compensating participants). Someone would have to foot the costs of this.

2. Studies can be incredibly labor-intensive. PI's can get away with running studies that require thousands of man-hours because they have a captive market of PHD students, Post-docs, and research assistants all willing to work for low wages or for free. PHD students usually don't have the same amount of man-power.

3. For obvious reasons, studies that require high cost, high man-power work tend to get replicated naturally less. In other words, the least practical studies to replicate happen to also be the most necessary to replicate.

A couple of things I would dispute:

> it seems unlikely that replication studies would be devalued merely because they're done by students

I think academics value work in a particularly skewed way. There is "grant work" and there is "grunt work". Grant work is anything that actively contributes to getting grants for one's institution. Grunt work is everything else. PHD's can do grunt work, but that doesn't mean it will be valued on the job market. For example, software development is actively sought after in (biology) grad students, because it's a very useful skill. However, I've also seen it count against applications as professors because it shows they spent too much time on "grunt work". Software development skills don't win grants.

> Few students want to stay in academia

In some fields there aren't any options except to stay in academia or academia adjacent fields.

Reply on news.ycombinator.com