Hacker News Reader

Comment by jgrahamc | Hacker News Reader

I wrote to the lead author asking them to release the source code of the software that they used for the data analysis. I have yet to receive a reply, but it's pretty important to eliminate a programming error from this.

For example, there were small errors in the Met Office's climate change software that I detected. The scientific papers were correct, but the translation into code was not. This could have happened here and it would be better if they released the code.

http://blog.jgc.org/2010/02/something-odd-in-crutem3-station...

View on HN

Asking for the source code of the OPERA analysis is similar in scope to asking for the source code to gmail. Unless you made a very specific request and explained how you would be qualified to review it, I doubt they are taking you very seriously.

Furthermore, I doubt anyone at OPERA is in a position to even fulfill your request. The minimum number of separate systems with exclusive sets of code is three, and probably more than ten.

However, physicists do make extensive use of unit tests, so that may be some consolation.

> However, physicists do make extensive use of unit tests, so that may be some consolation.

Ehm, well some do. And those that do, endlessly complain about most of them that don't.

Fact of the matter is, most physicists aren't trained in proper software engineering methods. Quite a few, but not nearly a majority are very interested in computers and computing so they may pick up a thing or two or actually bite their teeth in doing things the right way, and then there's most of them who love Physics for being Physics and Science for Science and see the computer as just another tool next to their pencil and paper, and just "get things done" and screwed be the next PHD that has to pick up on their research.

At least, that's what I hear from a friend doing a PHD involving analysing huge datasets coming from apparatus measuring high energy cosmic particles.

Of course the LHC is a much more prestigious project, so you might expect somewhat higher quality, but you can't change the culture. Physicists are generally not Software Engineers.

I find it very strange that the source code is not routinely released in these circumstances. What possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely?

At the very least it should be as available as the papers themselves.

The only advantage I can see is that when the theory, formulas and algorithms are published, not the specific implementation itself, it forces people who want to who want to verify your work to write their own implementation. It kind of makes sure that a bug in your own implementation doesn't pollute the results of those trying to verify your work.

Exactly. It shouldn't matter if Lab A implemented the algorithm in Matlab and Lab B implemented the algorithm in NumPy.

Take the data, apply the described algorithms, get the same result. If Lab B doesn't get the same result, and Lab B has confidence in their implementation, then there's probably something funny with Lab A's code or data.

Reusing Lab A's code on Lab A's data just reproduces Lab A's errors, if any.

This is very true.

However if I can just look at Lab A's code and spot an error in it, that will allow me to discount their results without having to redo their experiment, potentially saving years of work and millions of dollars.

Maybe that's what researchers are really afraid of?

" discount their results without having to redo their experiment, potentially saving years of work and millions of dollars."

Or you could just analyze their data using their described methods and your own implementation.

The problem is more complex. Some errors will damage the scientific conclusions, others will show up as fractions of a percentage point inside a much larger confidence interval, and will ultimately not matter much.

In machine learning research, for example, most evaluations consist of running a new program on soem data, getting results back, and from these results computing some aggregate measure of performance. A bug on the code that computes this measure of performance is _really bad_ and can invalidate all your conclusions. If that code is right, however, a but on the code that trains your model is completely meaningless, because as long as your results are good you can argue that you actually meant to write a paper about the model actually implemented rather than the model you were supposed to implement.

I'm sure other scientific areas have similar distinctions, and a naive code reader might fail to notice that a bug is harmless (and there's also the fact that scientific code carries within it a lot of assumptions about the data which, if broken, can be buggy, but are not broken by real data).

What possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely?

Laziness, lack of time, and the fact that most scientific code is not suitable for public consumption. For instance, my code includes error messages such as "What the fuck????" and "This never happens" which I'd need to take out in order to prepare it for publication.

In addition, there's the assumption that source code is a pretty trivial implementation detail; we publish the algorithm but not the details of the implementation, just as the experimentalist tells us what he built but doesn't tell us what brand of screwdriver he used to build it.

The thing is though, as I'm sure you're aware, the algorithm in abstract is utterly irrelevant. The algorithm as released is never actually executed, it is the implementation that gets executed.

One "trivial" detail in the implementation, one tiny little floating point rounding error, can throw the entire thing. We all know this. So why is the scientific community so unwilling to face this elephant in the room? And why is nobody else confronting them about it? Of course there's jgc, with his admirable record of confronting people about things, but that's the only example I can think of.

You would've thought the "Al Gore made up AGW to devalue my exxon shares" angst-brigade would've jumped on this, but they seem more concerned with digging through departmental email gossip.

"The algorithm as released is never actually executed, it is the implementation that gets executed."

If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base.

A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.

Still, this is not a good reason not to release code. We can deal with funny comments, bad jokes and dead code. Why not let people examine the code? They are not programmers who will be judged by the neatness of their code.

What if they don't have code, because they're using a commercial product to do the recording?

Well... That part has to be certified as a black box, but even if your code is a Mathematica notebook, it would be useful to have it published along the raw data.

Again, there is no reason to be embarrassed of comments like "this should never happen". My own code has parts that raise "ThisShouldNeverHappenError" when something that should never happen disregards my opinion and happens anyway. ;-)

You've implicitly accepted that the first implementation is correct, which I've found is one of the mental blocks that people defending this practice seem to have with frequency that surprises me. It isn't true. It isn't even the sort of thing you can try to debate your way out of, because it simply isn't true. Rather complicated numeric code written by non-software engineers is not something I'm willing to presume correct. I wouldn't presume it correct if it were written by software engineers, either, but at least then we'd probably have some sort of halfway reasonable testing to point at.

Further, the Kolmogorov complexity of a simulation is typically a wee bit larger than the Kolmogorov complexity of a square root implementation. That metaphor is beyond useless, it's deceptive.

"You've implicitly accepted that the first implementation is correct"

Not at all. The questions are 1) does the method described sound reasonable and 2) if I implement the same methods, with my toolset, and process the same data, do I get the same result?

If the results are different, using different implementations of the same methods, then it's possible that either party could have the error. But if the second party is reusing well-tested code of their own, or something from Matlab or whatever, then it is more likely that the unknown code of the original lab is the problem.

You might have a point about a simulation, but an awful lot of science isn't simulations, it's analyzing recorded data.

s/simulation/any real code/. It's still not a square root, which is a total straw man. If it were that simple, there'd be no code to write in the first place.

Okay, instead of 'square root' try spike sorting algorithms for recorded neuron activity. There are various ways of doing it, various software packages, free and commercial, ways of doing it in Matlab, etc.

(Ex: http://www.plexon.com/product/Offline_Sorter.html#Features)

Given a terabyte of recording data, I'd think it should be sufficient to specify the parameters they used for spike sorting. Someone else ought to be able to use a different package on the same data and obtain results similar enough to know if the first person fudged their paper.

Given a set of data, and a named algorithm, it ought to be possible to obtain the same result with any correct implementation of that algorithm.

That's begging the question. Yes, of course a correct implementation will by definition obtain the correct result. I'm questioning the casual way people assume they've got the correct algorithm implemented at all.

Is your paper nothing but off-the-shelf "spike sorting", or is that used as a component of something larger? Rolling back around to the original point of source code release being a desirable thing, if it's the latter, just knowing that this one library was used from what is probably literally a single sentence in the paper that reads "We used spike sorting software X to obtain sorted spikes", when presumably the library has knobs whose settings you don't know and you still don't even have their raw data to check the settings with, you still know virtually nothing about what was actually done. Source release ought to be standard.

The degree to which people who are putatively scientists will go to bat to defend making it difficult or impossible to replicate their experiments boggles my mind.

You're still obstinately missing the point in your nerd rage — letting other people run the same source code does nothing to replicate that part of the experiment — they need an independent implementation to do that! Releasing the source code publicly actively works against that goal because anyone who reads it won't be able to do a black-box implementation.

Besides, the code is almost always worthless compared to collected data. What would actually be most valuable and practicable is for groups to be running their proprietary code on each other's public data for confirmation.

This assumes errors by implementors are uncorrelated, which is almost certainly false. As programmers we often all make similar mistakes, particularly in exacting numerical code.

True, but the second implementation may use different, well-tested, mature code.

If the first lab used C++, and the second lab used Matlab and its libraries, and the third lab used trusted code based on NumPy, I would be surprised if their errors were strongly correlated. Brand new code? Sure. But not code they've used for several years through multiple experiments, or code that's part of a widely-used software package.

The thing is, labs working in similar areas most likely have ready, tested, trusted software toolkits for doing similar tasks, especially when it comes to data analysis.

It may seem to a programmer that the fastest, easiest thing to do would be to read the code, in fact the fastest, easiest thing to do is probably for another lab in the same field to do an independent analysis of the data with their own code.

"trusted code based on NumPy" <- (Just to note, a number of NumPy's routines have seemingly known numerical instability issues; I would not trust it, and was burned by it badly a few months ago. If you've used a specific function before and it seemed to work, then that's great; but if you haven't: watch out.)

Biology researchers don't get to reuse the other guy's mice, why should they reuse their code? As long as the paper describes the algorithms used, anyone should be able to duplicate the work using their own preferred implementation.

Also, a lot of the code may well be specific to the apparatus used, which may be unique to a lab, consisting of custom-built hardware.

Mouse can't be perfectly copied. Code can.

If biologists could copy mice from other experiment to give them their own drugs and compare results, they would surely do this along with using their own mice.

The more ways to compare experiments, the more sure we will be about results.

I understand cleaning up code is a lot of work, so don't do it, just release it as it is. I know it will be ugly, will include obscenity, etc. If somebody need it, it will do the rest of work.

"The more ways to compare experiments, the more sure we will be about results."

No. The more exactly you duplicate the original experiment, the more likely you duplicate any confounding issues. If someone observes something through a telescope, it's best to verify it through a different telescope somewhere else, to exclude the possibility of an optical quirk of the first telescope.

Some people will run the same code on the same data. They will get the same results, but so what.

Some other people will run the same code on different data (maybe from simulation, or when checking their own theory). They will get some other results, or the same.

Some people will recreate experiment from scratch, and get code B and data B.

Some people will run code A on data B. Some people will run code B on data A.

Now we can't do that. The code is already written, it's waste to hide it, it's not like publishing code prohibit other people from writing their own code.

That's how you check for errors - by changing as little as possible and observing results.

>Biology researchers don't get to reuse the other guy's mice

In a sense, they actually do.

https://secure.wikimedia.org/wikipedia/en/wiki/C57BL/6

https://secure.wikimedia.org/wikipedia/en/wiki/BALB/c

They use identically-specified mice. They don't use the same mice. The mice used by one lab could, theoretically, be off-spec mutants. Or the lab could have made mixed up which mice they used.

Saying you used a particular strain of mouse is like saying you used a particular algorithm on your data. Asking to use the exact individual mice used by another lab might not reveal an error such as mixed-up strains.

Sure. I wasn't posting those links to further this argument, I posted them simply because they are interesting.

Besides, they're only inbred strains, not clones.

They can also be pretty damned expensive. Like $400+ for a mouse.

Looks like I'm in the wrong business.

How does that come about? Patent shenanigans?

Probably. But really, if you're spending $50,000 on an experiment, you're probably more interested in getting mice that are really what it says on the label, than in saving a few bucks and possibly shooting your experiment and/or career in the nuts.

Pretty much anything 'lab grade' is going to be expensive. The lab I worked in ordered a lab grade laser for optogenetics experiments, and that cost something like $18k. Presumably that gets you a very precise, stable wavelength.

If I were involved in this project, I'd be sitting around saying "please don't let it be my fault, please don't let it be my fault".

Somewhere, somebody screwed up... and if the problem is in my code then I want to find it and correct it myself, rather than have it found and corrected by some random stranger who emailed me, and will then post the bug all over the internet so the world can point and laugh at the exact line of my source code that sent the world on a wild goose chase for FTL neutrinos.

So I don't blame this guy for not sending you his source code.

I don't blame him either. Random British guy writes to you in French and ask for your source code. Sure I'd be skeptical too (although personally I err on the side of releasing all my code).

The problem is that suppose it is a bug and suppose the code is being kept behind so they can find the bug themselves. A huge amount of time and money is wasted whereas a programmer could look at the code and compare with the paper pretty quickly.

Also in the case of high-energy physics its not exactly easy to duplicate experiments because of the eye-watering cost of the equipment.

"A huge amount of time and money is wasted whereas a programmer could look at the code and compare with the paper pretty quickly."

Maybe. It might be faster and easier to analyze the data using their described methods and your own code and see if the results are the same.

It's simple. For programmers, code is king. For scientists, data is king. Code can be recreated. Data is irreplaceable, so that's what is important, along with method.

Data depends on the code. You can't write neutrino arrival time to file on hard disk without some code in between the equipment and the file. That code can be wrong, too.

Some code has to be trusted, and scientists can't check every assumption every time, so I don't blame them. But why don't they allow others to check the code? It's the opposite of scientific method.

Perhaps an electrician should ask to check all the solder joints on all the cables?

How is that any different from a random programmer assuming the physicists are incompetent and presuming that what is really necessary is a code review?

If an electrician offered to check them for free, in his free time, why should they refuse?

EDIT: publishing code don't require dismantling equipment. It's not my fault electrician isn't very good analogy.

Because it would probably require dismantling vast amounts of equipment, and it would probably be detectable in the data anyway.

EDIT: True, not a great analogy.

Reply on news.ycombinator.com