Hacker News Reader

> ... doing the same thing twice is pointless.

That's what psychologists say (according to Richard Feynman in his now-famous criticism of the practice in "Cargo Cult Science"), but it's a fundamentally flawed view of science. Replication -- an exact duplication of a study and hopefully its results -- is a cornerstone of science. And without full disclosure of the original study's method and results, replication isn't possible.

> This is why the Hubble Constant started off being 150, then was 50, and now hovers around 69. Because everyone did the problem in their own unique way and didn't borrow each other's code and procedures.

That isn't an argument for the practice of withholding methods, it's an argument against it. Arriving at a realistic value for the Hubble constant wasn't accelerated by withholding methods, it was slowed. Imagine Einstein claiming that mass and energy are bound together in a clearly defined relationship, but not publishing the relationship or its derivation.

Arthur C. Clarke said, "Any sufficiently advanced technology is indistinguishable from magic." Apropos the present topic, a result unaccompanied by the methods used to produce it is also indistinguishable from magic, and certainly isn't science.

> If I wrote it, I want first dibs on analysing it, I don't want someone else taking the results I created and publishing them before me.

We're not taking about someone beating the originator to publication, we're talking about the originator publishing his methods along with his results and getting credit for both.

> Politics is surprisingly omnipresent here.

Perhaps, but always at the expense of science. Science relies on full disclosure -- both methods and results. How can a theoretical consensus be arrived at if the participants don't know what they're agreeing about?

"Replication -- an exact duplication of a study and hopefully its results -- is a cornerstone of science. And without full disclosure of the original study's method and results, replication isn't possible."

My first quarter in grad school in statistics, a girl who was a couple of years ahead of me quit. She was doing an internship with an MD, and every time he got a new diagnosed kid in the project he pondered which group to put the kid in. "This one's going to die, where do I put him to make the results come out right?"

The institution was considered second-class in that field, so he got the money to replicate a Harvard result. If he got the right results it would make him look more competent and he might get better grants. So he was doing his best to fudge the results so they would come out right.

She tried to argue that he should do his statistics correctly and he disagreed. She was so upset that her work was pointless, and that her career would be pointless, that she quit completely.

Ideally scientists would be rewarded for doing open science correctly. In some ways that is not the case now, and we should look at ways to fix that.

How can average mediocre scientists get job security without keeping secrets? Perhaps we are giving too many people a chance to be professional scientists, so that too many of them must lose out?

What a sad account. What a negative outcome for science. And I'm sure it's common in many fields.

> How can average mediocre scientists get job security without keeping secrets?

By way of systematic changes, not by anything they necessarily can do themselves at the level you're describing.

By way of people near the situation willing to speak out, like those who brought Jan Hendrik Schön down:

http://en.wikipedia.org/wiki/Sch%C3%B6n_scandal

Or in a similar way, by the actions of those who brought several highly regarded psychologists down:

http://science.nbcnews.com/_news/2013/02/20/17032396-scandal...

Title: "Scandals force psychologists to do some soul-searching"

Quote relevant to the present topic: "For instance, right now there's no incentive for researchers to share their data, and a 2006 study found that of 141 researchers who had previously agreed to share their data, only 38 did so when asked."

But the existence of the linked article and others like it means something is being done -- many psychologists have been outed as frauds, or fired, or forced into retirement.

So there's reason to hope that, if enough daylight shines on these practices and laboratories, things will change.

> "If you are publishing articles with no source code, how is that even science?" - We have a methods section in the paper, just like in the science fair.

yes but the pipeline goes METHODS -> (CODE) -> RESULTS.

you can read the methods and they check out. fine. you can't see the code. the results show something new and amazing. how do you know this is because of the methods used or because somebody flipped a minus sign in the code?

well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found completely different results, that are also new and amazing. who is right? the answer is: NOBODY KNOWS, because nobody is publishing their code!

so astrophysicist number 3 comes around, wants to know who is right, and has only these methods to go on, but no code. he has to start from scratch. because the universe hates us, he will find a third set of completely different new and amazing results that aren't even of the same datatype as the first two--but nobody knows who is right, because everyone is hiding their code and claiming it's based on the same methodology.

> doing the same thing twice is pointless.

No that's science.

What is pointless is doing the same thing twice, in the hope you're doing the same thing as the last guy, because you don't know what the last guy did. Also, hoping that if you make errors (like the last guy undoubtedly did, because hey this is unreviewed closed source code we're talking about), that they'll be different errors. Except you can't know, because nobody publishes their code.

All of this reminds me a little bit of the parable of the Stone Soup. And if the above behaviour is causing people stress and grey hairs about their job security, well I don't wish that upon anybody, but hoarding source code isn't the way forward either.

> well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found completely different results, that are also new and amazing. who is right? the answer is: NOBODY KNOWS, because nobody is publishing their code!

No one is going to spend time auditing the code to find a mistake either. In academia the way these things are, and should be, validated is by comparing the output to known results. In simulation this means do my simulations match experimental reality. In data analysis this means you check that you can verify something that's known to be true about a particular dataset.

That kind of validation in itself doesn't happen very often (it's often missing for example in computational chemistry). But in all honesty having the code, wouldn't (and doesn't) help.

I'm not arguing that the code shouldn't be open source, but that if you're presenting "new and amazing" results the way you back those up is not by saying you checked your code really well, but by showing that your method and implementation are consistent with known facts while presenting something new.

> you can read the methods and they check out.

I'm a materials scientist specializing in microscopy and image analysis, not an astrophysicist. In my area of interest, there is rarely sufficient information published to confidently repeat results. Indeed, the current reproducible research community had origins in computer vision (and was then embraced by the biostatistics community). I applaud the expectation of openness - especially if funded by government grants. Too often, we taxpayers pay for research that get trapped behind paywalls.

I agree the the academic astrophysics community has it tough. This is symptomatic of the entire academic community that is producing too many Ph.D.s desiring too few academic positions. Guess what - it will only get worse when the unsustainable government spending and debt causes the education bubble to burst.

> …hopefully they do it in a different way, doing the same thing twice is pointless.

I believe Newton said, "If I have seen further it is by standing on the shoulders of giants.", not, "I have rediscovered everything in a slightly different way because no-one gave me more than a vague method to go on"

It is no bad thing to have several independent implementations of an algorithm. If there is a problem with one, then the others are likely to show that there is a problem. However, without open source code, all you can do is say, "Mmm, something's wrong somewhere" and write another version. You may end up with several papers that agree and one that doesn't, but you still can't draw any conclusions without looking at the code. Ultimately, you get a situation where everyone has to repeat the same work and see what they get, when what they should be doing is poring over the original soources and discussing which bits could/should have been implemented differently and how that affects the result.

The code is part of the method. If you can't show us the code, don't expect me to believe your 'results'. That is science.

Newton also said, "If others would think as hard as I did, then they would get similar results." This seems to be part of the attitude OP is taking, that if others want his best, most unique results, they should arrive at them independently. But then when someone did that with Newton (Leibniz), the resulting fallout "redounded to the discredit of all concerned." (Here is a nice summary of the beef: http://www.ams.org/notices/200905/rtx090500602p.pdf)

I once looked and could not find any evidence that Newton said that. Since the quote is designed to sound good to modern ears, the attribution to Newton is probably bogus.