Hacker News Reader

Comment by scott_s | Hacker News Reader

Another conclusion is that scientists gain very little by publishing their code. In research terms, they're evaluated on published papers, receiving grants and graduating students. The incentives just aren't there.

View on HN

The article is just really poor, and mostly shows the author's complete lack of knowledge on how science is done, and even what it is about. One the biggest reason for not opening your code is to avoid given an unfair advantage to other researchers: academic research is extremely competitive, and the risk for free ride is significant.

I am all for opening tools more, but I don't think reproducibility is the reason for it. In any case, reproducibility does not mean everyone using the same software (with the same bugs) - it means independent experiments.

If they are willing to be honest, they can publish the code in the condition it's in. What's the extra work in uploading a tarball or exporting a single version to a site like github or Sourceforge? The paper (and the data, which should be published as well) provides all the context needed.

I suspect the extra work involved in publishing code is because the code is in such bad shape that it would decrease confidence in the work that depends on it. If that's the case, then it's all the more important that researchers establish it as a standard, non-optional part of publishing research.

Again, there's no incentive. You're arguing that it's not a burden, but you're not presenting any benefit it has to the researchers. If you want researchers to start doing this, then you need to figure out how to incentivize it.

Sure I am. There's general disrespect in academia for work that is incompletely reported in a way that hinders review and reproduction. It's a historical accident that refusing to publish source code has come to be treated as a special exception to the principles of good scientific practice. The more the basis for that exception is undermined, the more justified researchers will feel in demanding source code when they're skeptical of a result.

And here's the key incentive: lack of source code can be used as an excuse for ignoring and devaluing competitors' contributions. Scientists may be reluctant to raise a standard that would result in more work for everybody, but they can't resist a means of denigrating their rivals' work. Once the cat is out of the bag, scientists with weak reputations and disruptive results will be forced to publish source code to prevent their work being round-filed. Established scientists will find it untenable to hold themselves to an explicitly lower standard than everybody else, so they will eventually be forced to start publishing their code, too.

It's not a refusal, it's a neglect. What you're saying is, "If the culture was different, the incentives would be there." But the culture is not different. You can't just wave a magic wand. Changing culture is hard.

I explained how the culture will change. It will change because academics will use failure to release source code to throw doubt on the results of their rivals, and people (especially those who aren't already established in their fields) will answer that criticism by releasing code, which will raise the standards for everybody else. There's disincentive to be first, because everybody will pretend to be shocked at the buggy, undocumented code, but not everyone writes bad code and not everyone will feel able to refuse.

You want me to actually change the culture right now in a comment thread on Hacker News? I think predicting how it will happen is the best I can do, sorry.

You explained the manner in which it would change, not why it would change. That's the hard problem.

If that's not clear, you did not explain why academics would "use failure to release source code to throw doubt on the results of their rivals" when they do not do that now. I don't disagree with you that this would be preferable to the status-quo, nor am I disagreeing that it would be sustainable once the change occurred. I just don't see any reason why the change will occur in the first place.

I think you're talking past each other. As I read it, dkarl is saying that we should encourage academics to do this. That is the implicit reason why they would do it — because we're pushing them to.

Are there incentives to have a lower/higher barrier for someone else to reproduce your findings and build on them?

No, not really.

If you have no reputation and results that contradict expectations, you definitely want someone to reproduce your work. Otherwise, people may follow a Bayesian argument and decide the mostly likely conclusion is that your work sucks.

Reply on news.ycombinator.com