Hacker News Reader

Comment by Blahah | Hacker News Reader

I'm a computational biologist. If I read a paper where the results of an analysis are presented, but the code used is not, that analysis is worthless. Thankfully, in computational biology & bioinformatics at least, open source is widespread and growing.

Every programmer knows that every programmer makes mistakes. Lots of them. Mistakes that can completely change the outcome of an analysis. And if the results of your analysis are to be taken seriously, people have to be able to trust them, which means they have to be able to check your working.

Not publishing your code is anti-science, selfish, and IMHO should be disallowed in the literature.

View on HN

"Not publishing your code is anti-science" ... "open source is widespread and growing"

I've noticed that few people realize that "publishing your code" and "open source" are two overlapping issues, but they are not the same and the needs of one field are sometimes opposed to the other field.

Open source necessarily implies that the people who receive the software have the right to modify the software, redistribute it with or without changes, and be able to do so for a charge.

No one has been able to tell me why software for scientific publications requires the third of those abilities. Everyone is agreed that access to the code and the ability to modify it makes it much easier to review and understand what it does. I can also understand the need to share modifications with collaborators, in order to help carry out the analysis. But I don't see why published software can't have a prohibition on charging a fee for using modified versions of the software, or services rendered which use the software.

My first question for you, then, is: do you need the ability to commercialize someone else's software in order to provide good scientific review of their publication? More specifically, what sorts of review would that prohibition eliminate?

In the other direction, I have scientific software which I sell for about $30K. (This is not hypothetical - I really do have this). Customers get the source code under the BSD license. This falls into every standard definition of "free software" and "open source" software. There's even an essay (at http://www.gnu.org/philosophy/selling.html ) encouraging people to sell free software.

If I publish a paper which uses the software, then am I obligated to give my peers access to the source code for no fee? Or can I publish the paper, say it's available under a BSD license, and charge $30K for access to it?

So my second question is, are there limits on what I can charge people in order to get access to my open source software, which was used for a paper? If so, what are they, and what is the ethical basis for that judgment? (For example, should it be "fair, reasonable and non-discriminatory"? Can it cover development costs? Distribution costs? Web site development costs?)

To clarify, I'm saying that publishing the source code so that others can run and modify it is the crucial thing. I'm not saying that not open sourcing your code is anti-science, but not making it available for scrutiny is.

So to answer your first question, I don't think the right to sell someone else's code is related to the scientific process. I can't think of a way it affects the progress of science to use (or not use) commercialisation-friendly licenses, except in some indirect economic ways I don't know enough about to predict.

As to the second question, I think if you tried to charge reviewers for access to your code, your paper would never be accepted at a journal. Science is largely public funded. When public funds are used for research, the fruits of that research should be made public. That includes your code.

Overall then it looks like you agree with me. The philosophical underpinnings of free software are different than the underpinnings of good science.

nods to your first answer. I would also say that being able to distribute source (with or without modifications) to others is important, though not as crucial. This prevents someone (cough Elsevier cough ACS) from having a monopoly on providing the source. But a restriction on, say, military use or wiretapping would not inhibit scientific progress, even if such a restriction makes it something other than free or open source software.

As for the second question, the scientific software I sell is not nor ever has been publicly funded. The people who have paid for it are all for-profit organizations. I don't even get R&D tax credits for it. While you're correct that much of science is publicly funded, my software isn't.

Going back to that question: are there limits on what I can charge people in order to get access to my open source software, which was used for a paper? If so, what are they, and what is the ethical basis for that judgment?

I can tell you that I would rather not publish a paper, in order to earn income selling my software, than to describe the techniques I used in making the software and the corrections and improvements to the existing literature that I developed. Which is better for overall scientific progress, and why?

This is especially important should I choose to publish as open access, since that journal in my field costs about $1,200, so I'm already planning to pay over a month's rent in order to publish. Should I also expect to lose the equivalent of a year's salary, in the hopes that publishing the paper helps as big enough advertisement to my services?

Yours is an interesting situation. Without public funding, in my opinion you have no moral obligation to provide free (as in beer) access to your software. Indeed, the greater social benefit (if there's a binary choice between nothing being published and there being a paper but no free (as in beer) source code), is gained by you publishing a description of the work. Others can then at least benefit from your theoretical advances. To directly answer your first question: I don't think there are any ethical limits on what you can charge (but there are economic ones).

Which outcome is better for overall scientific progress depends entirely on what your software does, how large the need for it is, etc. However, one thing that is almost certainly true is that providing free (all senses) access to the software will maximise the social benefit. So if you can find a model that allows you to profit whilst still doing this, I encourage you to do so (see last point).

If I were in your situation I would say the issue of whether or not to publish is economic: publishing a paper about the software will bring it to the attention of the scientific community. That should lead to increased custom for you, provided the software is good and priced appropriately. An example of this is Robert Edgar of http://www.drive5.com. He has written several pieces of software which have really advanced computational biology, especially USEARCH. He makes the 32-bit version available freely, and there's a ~$800 per machine license for the 64-bit version. He also makes some tools available freely. This mixed model seems to have performed very well for him, as he is very widely cited which gets him a high profile and many large bioinformatics institutions buy licenses for his software.

Again, whether publishing open access is a good idea depends on which journal you publish in, how many people will benefit from learning of your advances, etc.

Since you say all the people who have paid for your software are for-profits, you could consider an unrestricted academic license with a paid commercial license (similar to all the baseclear products: http://www.baseclear.com/landingpages/basetools-a-wide-range...).

"I don't think there are any ethical limits on what you can charge"

I think you can see that some others hold a different viewpoint from you and me. I've heard people say that the ability to review the software is essential for science and that the software must, in all cases, be available for trivial if not no cost, in order to allow that review. (I think this view doesn't have a moral justification.)

"the issue of whether or not to publish is economic"

Absolutely. It's advertising. It's then a question for me to decide how to maximize my profits AND maximize improvements to the field. (Only somewhat apropos, I always loved reading the PNAS blub for each article: "The publication costs of this article were defrayed in part by page charge payment. This article must therefore be hereby marked "advertisement" in accordance with 18 U.S.C. §1734 solely to indicate this fact.")

Getting back to the topic, suppose that a group uses 64-bit USEARCH to do research. You wrote "I'm saying that publishing the source code so that others can run and modify it is the crucial thing." However, that group is unable to provide the source code used to do their analysis. ("Licensee will not allow copies of the Software to be made or used by others", says the 32-bit license.)

If publishing the source code is crucial, then this secondary group, which uses the algorithm, is obligated to publish the software they used, no? Or does that obligation only apply to the primary developer of the algorithm? If the primary developer never publishes the source code, then should all secondary users be prohibited from using it in order to develop new science?

Does that prohibition extend to using Excel? Oracle? Built-in software in sequencing hardware? I don't see an obvious bright-line demarcation.

As for "unrestricted academic license", I disagree with some of the distinction between academic license/commercial license. There are academic labs with a lot more money than I have. The group I was in, in the 1990s, had a NeXT or IRIX box on each student's desk, for example. There are also academic labs which act as a front, of sorts, for a professor's commercial interests. Also, the software I'm working on makes things fast - some 40x more than what people would do on their own. Any group can make the time/money tradeoff, and an academic group may easily have more money than time.

I've decided on a different view. Anyone can get access to the older versions, at no cost and under a BSD license. The newer versions are available to anyone, for a fee, and under the BSD license. I'm not dependent on this product for revenue, so it's a test to see how successful this business model might be.

If the paper is based on the analysis of sequence data, it's almost certainly been processed by closed or "shared source" primary data analysis tools. That's true of Illumina/Roche/ABI. Biotech vendors have quite a long way to go in this regard.

In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis.

Out of curiosity, do you work for industry or academia?

That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result.

I'm in academia (see my profile for details).

It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods.

It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods.

It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.

I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.

Sometimes validation can be done without knowing the actual algorithm or being able to rerun it.

For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program.

Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible.

You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction.

A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important.

For those cases, I don't see the need for access to the underlying software in order to validate the result.

The problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny.

I agree about the genetic algorithm.

Reply on news.ycombinator.com