Comment by patio11 | Hacker News Reader

Comment by patio11 | Hacker News Reader

They fear what would happen to their reputations if it were obvious how the sausage was being made. (Not unique to climate change: my academic AI research produced decent results on some data sets with code of truly abominable quality. Then again, I wasn't asking anybody to bet the economy on my results.)

Plus, if people actually provided code, someone might actually get it into their heads to run it. That can't happen. No, literally, it can't happen. The level of professionalism with regards to source control, documentation, distribution, etc in most academic labs is insufficient to allow the code to be executed outside of the environment that it actually executes on. If you put a tarball up somewhere, somebody who tries to run it is going to get a compile error because they're missing library foo or running on an architecture which doesn't match the byte lengths hardcoded into the assembly file, and then they're going to email you, and that is going to suck your time doing "customer support" when you should be doing what academics actually get paid to do: write grant proposals.

This, by the way, means that peer review by necessity consists of checking that you cited the right people, scratched the right backs, and wrote your paper in the style currently in fashion in your discipline, because reproducing calculations or data sets is virtually impossible.

If your process isn't sufficiently documented so that your professional peers could reproduce your results with your data, then your process is unreliable. We berate academics all the time for improper or incomplete documentation of manual experiments. I can't see any reason why source code should be held to a lesser standard.

Well, really it's the algorithm that needs to be published so others can reproduce the results. In principle, publishing code is similar to publishing the sequence to put together your experimental apparatus, which isn't done either.

I think pretty much everything looks more impressive before you see how it actually works.

Seriously. Relatedly, it is seriously impressive that systems this comprehensively screwed up seem to still converge on producing acceptable work much of the time. Big companies manage to get us all fed and fly us around the world. That scares the bejeesus out of me. I have put my lives in the hands of someone who was selected by an HR department (assisted by, even worse, a union)

Convergence of scientific results could be validation of the theory. But it could also be because people are anxious to publish contradictory results.

From Richard Feynman's1974 CalTech Commencement:

"We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease.

But this long history of learning how to not fool ourselves--of having utter scientific integrity--is, I'm sorry to say, something that we haven't specifically included in any particular course that I know of. We just hope you've caught on by osmosis"

anxious to publish

Should be "anxious not to publish", or "anxious about publishing", I think.

I get your point, but those results were still diverging from the original wrong answer and converging on the right one, just not di/converging as fast as would be ideal.

I don't know about that - I've seen a lot of industrial/manufacturing processes over the years that were damned impressive and had lots of complexities that I had no idea existed.

The only exception that I can think of is GPS (not the receivers, the system). Most people simply know that it involves satellites somehow. That system delights me in its cleverness, effectiveness, and scalability.

Nuclear magnetic resonance, as used to make MRI scans, is pretty damn impressive if you ask me.

The first time I looked at the code for a large 3D game engine, I felt this. It was almost like someone explaining a magic trick for you, you go "oh, so it all just worked because I was looking from this direction and that detail is something most people don't notice". Very disappointing, and it actually ruined it a bit for me.

I don't buy the point that once source code is published, you have to do all the "customer support" stuff. You don't have any obligation to do customer support so that you should be able to decide when you'd like to reply the request or whether reply it at all.

Source code is a very precise way to exhibit the work you have done. It is so precise that even particular machine can understand it. I am really happy to see that recently in NIPS more and more people published their MATLAB code along with their papers.

Oh, you must do "customer support"! Otherwise, they will use your code and report 1) that your code does not work or that 2) their code is better because they did not use your code well.

And for code that is not easy to install (e.g., requires many libraries, frameworks, dev tools, etc.), you need to spend time on documentation and even then, you often need to answer questions like "how can i create an eclipse plug-in to use your code or lxml failed to install"...

Mostly you only concern about the criticism from within scientific community. Since your result is published, generally, people with background will spend time to figure out how to use it by themselves. And their criticism is only relevant when it gets published. At that point, you are surely obligated to response such criticism. Most of the time, it is really not relevant that some random people send you email and have such "eclipse plug-in" problem.

People with background usually don't have the time to really discover how it works to make it run. They'll just ask.

"some random people send you email and have such "eclipse plug-in" problem" -> They are not random people, they are Ph.D. students working on their thesis and you can be assured that they will find a way to publish their comparison one way or the other. Some Ph.D. students will go at great length to figure out other's code before asking a question, but some will suck your energy big time and still do a poor job at representing your work. Now compare this to those who do not release their code: they can only be evaluated on their published results and they can spend their time on their next study/research work.

My point is that there is not a lot of incentive to release code: it requires a lot of your time, it is not always clear that granting agencies/employers consider this as being more positive than publications, and it exposes your work to all sorts of unfair replication/comparison.

Now, if research work without released code were never cited, that would be a terrific incentive to release your code :-)

P.S. I speak from experience and I release the code of my most important research work.

"Mostly you only concern about the criticism from within scientific community"

Ha. That doesn't apply to climate scientists. And there was the case of Andy Schlafly of Conservapedia (a creationist lawyer) demanding all sorts of nonsense from a scientist who documented evolution through many, many generations of bacteria.

At its worst, you get ideological cranks ganging up and demanding information, possibly through FOIA requests, pretty much for the sole purpose of using up all the scientist's time responding. Then when they don't respond, and/or it turns out they aren't subject to FOIA requests, the cranks freak out, break into a server, steal emails, and declare a conspiracy.

"Otherwise, they will use your code and report 1) that your code does not work or that 2) their code is better because they did not use your code well."

This x 1000! This is a topic near to my heart as I've actually had people use models & software we provided to them, not making an effort to understand fundamental concepts and theory behind it (or just being incapable of understanding) and then publish papers saying our models are crap! (not even providing better ways, nor comparing to anything else, just saying 'I put in some data and the results were unrealistic, this model is bad).

I agree. I've released my code, and there are several examples of people taking it and using it to do stuff that is totally outside its range of reliability because they just ran the code and didn't bother to learn enough about the problem. Then they publish crap and the crap comes back to me because MY code gets a bad rep for creating incorrect results.

"Then again, I wasn't asking anybody to bet the economy on my results."

To be fair, no climate scientist is asking anyone to bet the economy on their results, it's the various interest groups that pressure scientists into giving an answer that is less nuanced as they would have liked and then blame them when things go wrong. (at least for the 10 or so I know personally, but I'm pretty sure no others do, either). Actually this is true for all scientists I know in field that have political relevance.

Publicly available statements and evidence of past behavior has led me to reject my default charitable supposition that climate scientists and interests groups are disjoint sets.

I've attended climate science conferences and worked for years writing code for and with climate scientists. Down in the trenches, pretty much the only thing I saw was scientists interested in accurate results. I don't, however, claim to have worked with nearly everyone in the field. But I do on occasion feel compelled to defend my former coworkers (I work for myself now).

Can you link some examples?

Well I would nuance that, in that there are many interest groups why try to sway public opinion by posing as scientists. Like ANH I write code for and have worked with climate scientists, actual ones who have no real political or financial motivation and just really try to model the world around us as accurately as possible. So I guess it boils down to the definition of 'scientist'. I don't call every quack who get a book on Lulu or marketer who is paid to write scientific-sounding 'position papers' a scientist.

How is that necessarily a charitable supposition? If a scientist believes in his or her research, it would be positively inhuman of them NOT to advocate for the application of it.

This. Especially in a field where, if the questions are to be taken at face value then the results can be incredibly socially important.

I am optimistic that more codes will become available soon. This subject is dear to the heart of many scientists and is often discussed -- see this recent HN item:

http://news.ycombinator.com/item?id=1789134

Reply on news.ycombinator.com