In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis.
Out of curiosity, do you work for industry or academia?
Out of curiosity, do you work for industry or academia?
I'm in academia (see my profile for details).
It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods.
It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program.
Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible.
You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction.
A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important.
For those cases, I don't see the need for access to the underlying software in order to validate the result.
I agree about the genetic algorithm.