People not liking open source (and it's not Oracle) | Hacker News Reader

People not liking open source (and it's not Oracle)

weblogs.java.net

30 pointsGarbage24 comments

in my experience, it's common in academia for code to "float around". often it's not officially open source or distributed, but you know someone who has the code for some paper, so you can use it. there's even a few textbooks in this state (for years).

i think the real reason for not releasing this code is time---you might be surprised at just how much effort goes into making a paper/talk/article polished. sure, you could just release the hacky code you used to do your calculations, but you'd like so much more---for it to be useful, for it to be a credit to you, for it to also do XYZ. research code is efficient in that it gets exactly what you want for the paper; it's rarely usable by anyone else. there is essentially zero payoff in academia in publishing code, and so there's no time to fulfill these desires.

I definitely agree with this. Besides, the code itself is an insignificant part of the process. Any paper will include more than enough evidence to back up its assertions and provide a path for duplicating its results.

Exactly. In my experience, if someone wants to look at your code, they will e-mail you and ask for it. Then you e-mail it to them.

Sharing on something like sourceforge just simple isn't done. It's not because scientists are "cheating," it's because it's not part of the culture.

There could be some payoff for releasing code: If other people use your code, you could probably get a citation for it.

Another conclusion is that scientists gain very little by publishing their code. In research terms, they're evaluated on published papers, receiving grants and graduating students. The incentives just aren't there.

The article is just really poor, and mostly shows the author's complete lack of knowledge on how science is done, and even what it is about. One the biggest reason for not opening your code is to avoid given an unfair advantage to other researchers: academic research is extremely competitive, and the risk for free ride is significant.

I am all for opening tools more, but I don't think reproducibility is the reason for it. In any case, reproducibility does not mean everyone using the same software (with the same bugs) - it means independent experiments.

Are there incentives to have a lower/higher barrier for someone else to reproduce your findings and build on them?

No, not really.

If you have no reputation and results that contradict expectations, you definitely want someone to reproduce your work. Otherwise, people may follow a Bayesian argument and decide the mostly likely conclusion is that your work sucks.

If they are willing to be honest, they can publish the code in the condition it's in. What's the extra work in uploading a tarball or exporting a single version to a site like github or Sourceforge? The paper (and the data, which should be published as well) provides all the context needed.

I suspect the extra work involved in publishing code is because the code is in such bad shape that it would decrease confidence in the work that depends on it. If that's the case, then it's all the more important that researchers establish it as a standard, non-optional part of publishing research.

Again, there's no incentive. You're arguing that it's not a burden, but you're not presenting any benefit it has to the researchers. If you want researchers to start doing this, then you need to figure out how to incentivize it.

Sure I am. There's general disrespect in academia for work that is incompletely reported in a way that hinders review and reproduction. It's a historical accident that refusing to publish source code has come to be treated as a special exception to the principles of good scientific practice. The more the basis for that exception is undermined, the more justified researchers will feel in demanding source code when they're skeptical of a result.

And here's the key incentive: lack of source code can be used as an excuse for ignoring and devaluing competitors' contributions. Scientists may be reluctant to raise a standard that would result in more work for everybody, but they can't resist a means of denigrating their rivals' work. Once the cat is out of the bag, scientists with weak reputations and disruptive results will be forced to publish source code to prevent their work being round-filed. Established scientists will find it untenable to hold themselves to an explicitly lower standard than everybody else, so they will eventually be forced to start publishing their code, too.

It's not a refusal, it's a neglect. What you're saying is, "If the culture was different, the incentives would be there." But the culture is not different. You can't just wave a magic wand. Changing culture is hard.

I explained how the culture will change. It will change because academics will use failure to release source code to throw doubt on the results of their rivals, and people (especially those who aren't already established in their fields) will answer that criticism by releasing code, which will raise the standards for everybody else. There's disincentive to be first, because everybody will pretend to be shocked at the buggy, undocumented code, but not everyone writes bad code and not everyone will feel able to refuse.

You want me to actually change the culture right now in a comment thread on Hacker News? I think predicting how it will happen is the best I can do, sorry.

You explained the manner in which it would change, not why it would change. That's the hard problem.

If that's not clear, you did not explain why academics would "use failure to release source code to throw doubt on the results of their rivals" when they do not do that now. I don't disagree with you that this would be preferable to the status-quo, nor am I disagreeing that it would be sustainable once the change occurred. I just don't see any reason why the change will occur in the first place.

I think you're talking past each other. As I read it, dkarl is saying that we should encourage academics to do this. That is the implicit reason why they would do it — because we're pushing them to.

I think it's worth pointing out that taking someones code and running it again on the same data does not constitute an independent verification of the original work. Just like scientists don't go using the same lab to verify an experiment, they shouldn't use the same code.

That said, it is often useful to release code for other reasons. For applied areas, releasing the code in the form of a software tool is fairly common. This allows people from the target area, who may not be computer scientists, easy access to the new method.

Code review seems to be nearly non-existent in the scientific community, although it does happen occasionally when one researcher uses another's code base for further experimentation.

Investigating the world through computation is a relatively new phenomenon for science.

Science is discussed and advanced through papers. These papers aren't an exact recording or rendition of what was accomplished, but a discussion of the idea, methods used, and eventually a conclusion. Methods may be algorithms, but it doesn't necessarily mean the actual source code.

The idea is that the actual code that a scientist used isn't terribly valuable. If another scientist wants to verify the conclusions, they should just whip up another bit of code (or, do the same experiment) themselves.

Asking for scientists to contribute source code is not wrong, it just isn't done (to my knowledge) with other modes of scientific exploration.

This isn't something new at all! Chemists have been using computers extensively since the 70's, and this can be evidenced by the fact that several physical chemists at my university still use Fortran. Computers have been used in science from their inception. Only now are they becoming indespensible for molecular biologists--but they still have a long way to go in both terms of parallelism and the related tools to be really practical for the intense intermolecular interactions we deal with.

Why don't we publish our code? Do you publish one-off shell scripts or database migration code? The only important code is already open source or is (unfortunately) commercial in nature.

I regularly see dotfiles in github repositories.

Maybe there is a barrier to adoption in the form of git. A more polished version of Gist - marketed to scientists and researchers - would increase uptake.

What do you mean by "see[ing] dotfiles"?

It is new wrt the history of science, or more specifically the history of academic journals. I am aware of the history of scientific computation -- that's not the point.

The point is that scientists don't need to publish certain things, one of them being the source code that helped them discover their findings.

Shell scripts and database migrations aren't super interesting, but you would want them if you needed to replicate results. I'm not saying they're interesting as published research, but neither is that random piece of scratch paper you used while trying to understand what is going on in some biological process.

It seems like the problem is that non-scientists don't understand the way science happens, and that often the source code isn't something that you would include in an academic journal.

Should science be done differently? Maybe. But that discussion is different than "why wont they publish their source?"

> I can only conclude that many scientists are not confident at all with their theories, or they are purportedly cheating.

That's really not an insightful conclusion. I think it's rather a question of pride and effort: the code probably needs to be cleaned up before release, and the reward for investing that time is very low (career-wise). Researchers are going to publish their source code when it becomes required by most major journals and conferences.

I've worked as an academic hacker.

Amongst dedicated biologists, there's no motivation to release your source code. It's the job of a biologist to publish papers so that he or she can pull down the next grant. Anything that doesn't further this goal is going to get zero traction.

Publishing it doesn't help you get into Nature or Science. Sure, Sean Eddy (www.psc.edu/general/software/packages/hmmer/) is going to release his code, but he's a methods guy. The software is the focus of publication. The only way code is going to get published is if the journals refuse to publish results without code.

Reply on news.ycombinator.com