A paper’s central finding was based on polling that probably never happened
fivethirtyeight.com
fivethirtyeight.com
Imagine that a scientist designs a very intelligent experiment to test a plausible hypothesis - she has an excellent plan to analyze the results (pre-registered, to avoid p-value hacking) - and after doing the experiment, she finds no evidence of the effect that she was looking for. Why should the result (which is the only thing the scientist is not in control of) determine the reward given to the scientist? Funding should be given to investigate a particular phenomena, not to find high impact results.
Thankfully, I've seen very little of the latter, but I've seen quite a bit of the former. Even unconsciously, people are afraid to design experiments that would give a black and white answer and falsify a cherish theory.
People with lots of positive results are not necessarily better scientists. They are just as often less scrupulous and better salesmen - Peter Thiel noticed that as well.
Negative results are essential for a complete understanding of science and when they are repressed, complete access to the globally available data set is not available.
If I'm an ethically-dubious pharma or lobby group, I love this situation. I keep funding testing until someone comes up with a positive result by chance or p-value hacking - then fanfare that result all the way to the media and to the bank. (Negative results don't need to be seen)
If I'm a cynical scientist, I take the money and p-value hack until I get the results my funding body wants, so that I have access to more money in their next round of funding. Now I'm far more successful than a completely honest scientist.
In fact, the honest scientist will either seek to replicate what the cynical scientist did, or leave the profession.
That's how we're wired, we simply like success stories much more than that we like stories of failure.
An hypothesis isn't a product, falsifying your hypothesis is success - you've increased the pool of human knowledge (or increased the support for an element of it).
In business good and bad products can both lead to profit and can both lead to losses. Profit and loss can both be considered success and failure.
On your last sentence - I don't think it's as self-evident as it seems. We tend to like the successes that profit us but in general I'm not sure we humans are that congratulatory of other's success. Schadenfreude seems very popular.
Startups fail, although nobody involved likes it. Science experiments are supposed to FAIL, many times. That's what science is.
It's one thing to get data and then fraudulently change it to fit a story; at least then other people can go back and un-do the damage. It's a different thing to not do anything at all, completely make up data, and waste all that hard work.
As Peter Winker, an economist at the University of Giessen
in Germany who co-edited a book about poll fraud, put it in
an email: “If the faker would have been a bit cleverer in this
procedure, I doubt that the fraud would have been found this way.”Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result.
This case (and many others) failed 3) but everything else seems more than natural. Physicists smashing particles at the LHC were also trying to "reinforce their preconceived ideas" of how particles work. They have the standard model and they want to validate it where possible.
[0] http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i...
I also think it's a systemic problem of positive results being far more rewarded (in terms of funding, prestige, etc) than negative results, which gives even more incentive to bias the result (intentionally or unintentionally) towards the positive, satisfying outcome.
Imagine you have two social scientists, A and B. Scientist A publishes a positive result that resonates with people; it fits the social and political climate of the day. They give TED talks about it, and the result is reported in newspapers. Later experiments are unable to replicate the result, maybe because the original experiment was a little biased somehow, but nobody can say for sure and there is no scandal because there was no fraud. Scientist B is more rigorous in their experimentation and their work fails to demonstrate a hypothesis. They either publish the negative result, or don't publish at all.
Which scientist will have advanced their career more? I think that is the problem.
I think this is incorrect. A good experiment is designed to disprove the hypothesis. And so if the experiment fails to disprove, we've got one more vote of confidence in the hypothesis.
It's never possible to out-and-out prove any hypothesis. There's always going to be variables that we didn't think were relevant. All we can do is keep adding to the heap of evidence in its favor, by falsifying more and more objections to it.
Statistical tests are often designed around a null hypothesis that is defined as the opposite of the hypothesis we are testing. Once we can reject that null hypothesis we have proven the hypothesis. So for the LHC you'd disprove something like "there is no particle of energy X" (the null hypothesis) by finding a definite signal at energy X, thus proving that the Higgs Boson exists (the initial hypothesis). Prove/Disprove are relative terms though as you can only have a degree of confidence X on the validity of the hypothesis.
There are two problems. First is the easy problem which is that #3 doesn't always happen. The data gets slightly cooked, the study gets redone, explanations are given as to explain away the results. The way it is done can be from outright dishonest to quite acceptable.
But the bigger problem is that there are some hypotheses that are not thought up or tested, or are done so in low numbers. Namely ones that would be political/career suicide, but even those that could hurt funding or that would have biased a person from ever becoming a researcher. For example, say you wanted to test an experiment that gay women were more violent to their domestic partners that straight men, straight women, or gay men.
Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results.
>Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results.
Sure, and that may be the problem of looking for a predetermined outcome that haberman was referring to. If your testing methods are poor enough that spurious results are likely, then the fact that all your hypothesis are somehow biased is a problem. Though I'd say the problem is more with the testing methods than with the hypothesis.
Using the LHC example again no one could attack Higgs for being biased as an argument for the boson not existing after the LHC results were in. The provenance of the hypothesis should be irrelevant if the testing methods are good enough to provide strong evidence for it.
No. The critical difference lies in steps 2 and 3. You can never prove something in science, because you'll never get all the corner cases. What you're really after is designing an experiment that is able to falsify (=disprove) your hypothesis and run that. Because if you find one single instance that contradicts the original hypothesis, you know that your hypothesis in its current form is wrong, because it can't be applied universally.
So in the LHC case the hypothesis is "the Higgs exists and it's energy is X". Then they run the experiment and get evidence that says "there is a very high probability that there is a particle with energy X" and so their initial hypothesis is considered proven. If you don't like the word proven, then they conclude it's extremely unlikely to be false.
Nor is the rest of the comment much of an argument. Scientists have questions they want answered and to solve them they formulate hypothesis and test them. If you don't formulate hypothesis you have nothing to test and can't advance.
http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+...
Universities all over the world are working on repositories to store electronic resources such as data and code for the long-term. This is non-trivial, as data may need to be available for decades or even hundreds of years. Github, by contrast, has been around for 7 years.
Regarding the idea of organising research in a more change-control fashion, that's a more radical change. Academia will move slowly, and rightly so, as research practices evolve and persists for hundreds of years; to adopt an idea based on a (relatively, very) new method of software development is hugely risky.
---
It is interesting to note that as a programmer, I do not have that many issues with releasing partial or incomplete source code on Github, whereas as a computer scientist, I do not like to release partial or incomplete proofs at all.
It might be because the academic community cares about reputation: if you get a reputation as a researcher that releases incorrect proofs, your career is over. In programming, as the names are attached to project much more loosely, such negative reputation is rare and people care much less.
This was a relatively high profile paper. I wonder how much slips through in lower impact journals...
Sure, the paper's author may want you to know that it was all faked... but Science stands by what they printed, no matter what.
I'm okay with Science giving some due process here. Contacting the primary author and other parties, the peer reviewers, discussing it with their editorial staff, and so on. The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19. The odds are exceedingly high that a retraction will be issued.
Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing.
I have to disagree; I think jacquesm has the comment closest to the truth.
My comment is a minor distortion of the facts -- later in the article we get this:
> Science tweeted that it was assessing the retraction request and would in the meantime publish “an Editorial Expression of Concern.”
Here's the thing. Science isn't doing any research itself. Once a paper's author has admitted the data was faked, there is no role for Science to play; they can't look into the past and determine that it was actually real. They've acknowledged receipt of the message, and they're willing to print a notice that the paper is a lie -- but they don't want to retract it, because that would damage their record.
> The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19.
They have made a decision. They're going to print an unofficial warning. Without more pressure from the public, they will not print a retraction.
Remember, Green is the primary author. He contacted them; for them to contact him is somewhat superfluous.
Lacour has not indicated he supports the retraction request and it is extremely reasonable for Science to contact him about it and hear what he has to say before proceeding.
> They've acknowledged receipt of the message, and they're willing to print a notice that the paper is a lie -- but they don't want to retract it
"Assessing the retraction request" != "don't want to retract it".
> They have made a decision.
"Assessing the retraction request" != "have made a decision".
Green has retracted his paper by way of essentially accusing LaCour of falsifying data. His paper is retracted, fin.
Science has not yet decided whether it will on its own recognizance retract LaCour's paper.