At the very least it should be as available as the papers themselves.
Laziness, lack of time, and the fact that most scientific code is not suitable for public consumption. For instance, my code includes error messages such as "What the fuck????" and "This never happens" which I'd need to take out in order to prepare it for publication.
In addition, there's the assumption that source code is a pretty trivial implementation detail; we publish the algorithm but not the details of the implementation, just as the experimentalist tells us what he built but doesn't tell us what brand of screwdriver he used to build it.
One "trivial" detail in the implementation, one tiny little floating point rounding error, can throw the entire thing. We all know this. So why is the scientific community so unwilling to face this elephant in the room? And why is nobody else confronting them about it? Of course there's jgc, with his admirable record of confronting people about things, but that's the only example I can think of.
You would've thought the "Al Gore made up AGW to devalue my exxon shares" angst-brigade would've jumped on this, but they seem more concerned with digging through departmental email gossip.
If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base.
A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.
Again, there is no reason to be embarrassed of comments like "this should never happen". My own code has parts that raise "ThisShouldNeverHappenError" when something that should never happen disregards my opinion and happens anyway. ;-)
Further, the Kolmogorov complexity of a simulation is typically a wee bit larger than the Kolmogorov complexity of a square root implementation. That metaphor is beyond useless, it's deceptive.
Not at all. The questions are 1) does the method described sound reasonable and 2) if I implement the same methods, with my toolset, and process the same data, do I get the same result?
If the results are different, using different implementations of the same methods, then it's possible that either party could have the error. But if the second party is reusing well-tested code of their own, or something from Matlab or whatever, then it is more likely that the unknown code of the original lab is the problem.
You might have a point about a simulation, but an awful lot of science isn't simulations, it's analyzing recorded data.
(Ex: http://www.plexon.com/product/Offline_Sorter.html#Features)
Given a terabyte of recording data, I'd think it should be sufficient to specify the parameters they used for spike sorting. Someone else ought to be able to use a different package on the same data and obtain results similar enough to know if the first person fudged their paper.
Given a set of data, and a named algorithm, it ought to be possible to obtain the same result with any correct implementation of that algorithm.
Is your paper nothing but off-the-shelf "spike sorting", or is that used as a component of something larger? Rolling back around to the original point of source code release being a desirable thing, if it's the latter, just knowing that this one library was used from what is probably literally a single sentence in the paper that reads "We used spike sorting software X to obtain sorted spikes", when presumably the library has knobs whose settings you don't know and you still don't even have their raw data to check the settings with, you still know virtually nothing about what was actually done. Source release ought to be standard.
The degree to which people who are putatively scientists will go to bat to defend making it difficult or impossible to replicate their experiments boggles my mind.
Besides, the code is almost always worthless compared to collected data. What would actually be most valuable and practicable is for groups to be running their proprietary code on each other's public data for confirmation.
If the first lab used C++, and the second lab used Matlab and its libraries, and the third lab used trusted code based on NumPy, I would be surprised if their errors were strongly correlated. Brand new code? Sure. But not code they've used for several years through multiple experiments, or code that's part of a widely-used software package.
The thing is, labs working in similar areas most likely have ready, tested, trusted software toolkits for doing similar tasks, especially when it comes to data analysis.
It may seem to a programmer that the fastest, easiest thing to do would be to read the code, in fact the fastest, easiest thing to do is probably for another lab in the same field to do an independent analysis of the data with their own code.
Take the data, apply the described algorithms, get the same result. If Lab B doesn't get the same result, and Lab B has confidence in their implementation, then there's probably something funny with Lab A's code or data.
Reusing Lab A's code on Lab A's data just reproduces Lab A's errors, if any.
However if I can just look at Lab A's code and spot an error in it, that will allow me to discount their results without having to redo their experiment, potentially saving years of work and millions of dollars.
Maybe that's what researchers are really afraid of?
Or you could just analyze their data using their described methods and your own implementation.
In machine learning research, for example, most evaluations consist of running a new program on soem data, getting results back, and from these results computing some aggregate measure of performance. A bug on the code that computes this measure of performance is _really bad_ and can invalidate all your conclusions. If that code is right, however, a but on the code that trains your model is completely meaningless, because as long as your results are good you can argue that you actually meant to write a paper about the model actually implemented rather than the model you were supposed to implement.
I'm sure other scientific areas have similar distinctions, and a naive code reader might fail to notice that a bug is harmless (and there's also the fact that scientific code carries within it a lot of assumptions about the data which, if broken, can be buggy, but are not broken by real data).
Also, a lot of the code may well be specific to the apparatus used, which may be unique to a lab, consisting of custom-built hardware.
If biologists could copy mice from other experiment to give them their own drugs and compare results, they would surely do this along with using their own mice.
The more ways to compare experiments, the more sure we will be about results.
I understand cleaning up code is a lot of work, so don't do it, just release it as it is. I know it will be ugly, will include obscenity, etc. If somebody need it, it will do the rest of work.
No. The more exactly you duplicate the original experiment, the more likely you duplicate any confounding issues. If someone observes something through a telescope, it's best to verify it through a different telescope somewhere else, to exclude the possibility of an optical quirk of the first telescope.
Some other people will run the same code on different data (maybe from simulation, or when checking their own theory). They will get some other results, or the same.
Some people will recreate experiment from scratch, and get code B and data B.
Some people will run code A on data B. Some people will run code B on data A.
Now we can't do that. The code is already written, it's waste to hide it, it's not like publishing code prohibit other people from writing their own code.
That's how you check for errors - by changing as little as possible and observing results.
In a sense, they actually do.
Saying you used a particular strain of mouse is like saying you used a particular algorithm on your data. Asking to use the exact individual mice used by another lab might not reveal an error such as mixed-up strains.
Besides, they're only inbred strains, not clones.
How does that come about? Patent shenanigans?
Pretty much anything 'lab grade' is going to be expensive. The lab I worked in ordered a lab grade laser for optogenetics experiments, and that cost something like $18k. Presumably that gets you a very precise, stable wavelength.