Comment by lutorm | Hacker News Reader

Comment by lutorm | Hacker News Reader

Parent Full thread

The code I wrote for my simulations is not that big, but it's certainly big enough to be complicated and buggy. And it's GPL and available on my website.

And while I certainly don't expect any of my referees to go over the code line-by-line, one of the rationales I had for GPLing the code was to make it possible for someone to replicate my study. One of the basic tenets about publishing research is that articles should contain all the information needed to reproduce the results. This is not possible for simulation papers unless the source code is public (or maybe a binary is provided, but that has many pitfalls by itself).

Results generated with proprietary codes could just as well be hokus-pokus in my mind. It's not clear how it adds to the field unless others can reproduce and build on the results. And with reasonably complicated codes, it's not realistic to think that someone will implement their own version (though that is of course the absolute best thing).

So while I don't necessarily think scientific software should be peer reviewed, I'm very hesitant about results obtained with proprietary software.

What's your view on the mountains of scientific code that depend on proprietary math kernels and in some cases proprietary compiler platforms?

As you may guess, I'm unhappy about it. In astronomy, almost everyone uses IDL, which is a proprietary, closed-source language. (It also sucks, but that's beside the point.) I think this is bad, because it locks practically the entire research field into a closed platform. (Though GDL is working to be a replacement. I don't know how conformant it is, though.)

Still, I think this is less of a problem than the high-level scientific code. Most of the low-level functionality is pretty simple and (hopefully) much better tested than most scientific codes.

I'd just throw in that if someone uses your code to try to reproduce what you've done, they may actually just be reproducing a bug -- or in the sinister case, reproducing a faked result. So a more general point in this whole discussion would be that making the code available isn't always as important as making the mathematics or other theory behind the code available, and "reproduction" becomes "reimplementation", or some other corroborating element of the process as described by timr a few comments down (http://news.ycombinator.com/item?id=968456).

(Of course for large software endeavors that isn't practical either. So mattheww's point about trust unfortunately resonates.)

As for the proprietary issue, I think sometimes it might actually help reproducibility: if you can just drop a few lines of script into Matlab and examine them, it's potentially a much smaller task with a lot more battle-worn, common-ground code factored out than examining or re-implementing a full library of open code.

Reply on news.ycombinator.com