Social sciences suffer from severe publication bias
nature.com
nature.com
Real example: I'm a social scientist, and I have a hypothesis about a better method of analysing discourse (borne from theoretical musings, mostly). Once I spend the next 6 months testing that empirically, and it turns out to be nonsense or completely unfounded, I'm not sure how much value there is in putting it out there - a lot of social science hypotheses are (imo) quite ephemeral and move on quite quickly.
Part of that's a spillover from the fact that it's difficult to scale qualitative data gathering and analysis as an (often individual) researcher, and a frequent assumption that only quantitative data 'counts', but yeah. Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work.
There's a huge amount of value in sharing 'failed' experimental data and analysis (so long as the experimental methodology was sound). People can learn new things from meta-analysis of failures and correlating disproved theories against proven ones. In fact, there are journals dedicated to it; http://www.jnrbm.com/ and http://www.arjournals.com/ojs/ are examples.
Edit: A standard FM radio filter is +/- 90kHz around the tuned-in frequency, so if you choose 100.3 on your dial, you're really getting everything from 100.21 to 100.39 all at once. And 100.5 is really 100.41 to 100.59, so we're really only skipping 20% of the frequencies.
No, that is false, and further, it distorts entire fields. In psychology, for example, phony mental illnesses like Asperger's become established as real things only because of all the positive results, not accompanied by an equal number of negative ones that were never published.
And in drug therapies, many drugs are validated and sold only because there are only positive published results. In one correction to this trend, the FDA funded a study in which all studies of a given class of drugs -- published and unpublished -- were analyzed in a meta-analysis. The result was that the statistical results fell below clinical significance for the majority of subjects:
http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj...
"Drug–placebo differences increased as a function of initial severity, rising from virtually no difference at moderate levels of initial depression to a relatively small difference for patients with very severe depression, reaching conventional criteria for clinical significance only for patients at the upper end of the very severely depressed category."
Yes, and that argument is quite false. The history of science shows that positive and negative results have equal weight. But for that to remain true in modern times, the negative results must be published.
If you understood the history of science, I wouldn't needs to explain this. The Michelson-Morley Experiment had a negative outcome, a null result. It was replicated by people intent on "saving the ether" until everyone was satisfied that the null result was real. This ushered in the relativistic era and modern times. But if the null result had been discarded, it would have been an extraordinary disservice to science.
Penzias and Wilson were sure their antenna was dirty or their equipment was faulty, something pedestrian, until they systematically eliminated all such causes, and were left with a signal emanating from the sky in all directions. Having falsified all plausible causes, they consulted with some physicists who told them that they had detected the much-awaited cosmic background radiation. A revolution in cosmology and several Nobel prizes resulted.
Null results have an equal standing with positive ones. But if they're not published, the same studies will be repeated by people unaware that the path has already been tread. Also, conscientious scientists always remain alert to the possibility that their pet theory is nonsense and explainable in ordinary ways -- that's also a null result.
http://www.scientificamerican.com/article/social-sciences-su...
Quote: "When an experiment fails to produce an interesting effect, researchers often shelve the data and move on to another problem. But withholding null results skews the literature in a field, and is a particular worry for clinical medicine and the social sciences.
"Researchers at Stanford University in California have now measured the extent of the problem, finding that most null results in a sample of social-science studies were never published. This publication bias may cause others to waste time repeating the work, or conceal failed attempts to replicate published research. Although already recognized as a problem, “it’s previously been hard to prove because unpublished results are hard to find”, says Stanford political scientist Neil Malhotra, who led the study."
> Negative results can be minted at 100's per minute, by trying things that are very unlikely to work.
Yes -- like Edison trying to invent the light bulb. Edison generated thousands of null results, and one positive result.
"I have not failed. I've just found 10,000 ways that won't work." -- Thomas A. Edison
But repeating failed experiments has the same scientific value as repeating successful ones -- it solidifies the result, increase our certainly about it.
In real science, null results, and their replication, are extremely important. My favorite example is the Michelson-Morley experiment that expected to find differences in the speed of light due to the motion of the earth through the hypothetical ether.
The failure of this experiment, its publication, and efforts to replicate the result, also dutifully published, finally created a revolution in physics when Einstein explained why there was a null result, and coincidentally gave birth to the modern world.
I think the issue I'm trying to get at - I suspect there is less value in null proving a lot of social science hypotheses - the impacts are smaller than physical constants of the universe (or finding say, that the Higgs Boson didn't exist) - if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo.
I guess though, the main issue is: I'm sure as hell not going to get published saying it, which ties back nicely to the point made in Nature, really.
Yes, but that applies equally to proving positive social science results. My point wasn't to compare social science to other sciences, it was only to point out the value of replicating null results in general -- that it stands equal to replicating positive results.
> if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo.
Both findings have equal weight, therefore both findings deserve equal attention from replicators. By contrast, ignoring null results is a disaster for whatever science they occur in -- it may mean that other people might spend a lot of time spinning their wheels studying things that have already been found wanting.
Most research leads to no result. Most of the remaining research leads to only marginal results. Most of the research that leads to important results isn't still ground breaking. Only a tiny minority of the rare class of research that leads to important results is ground breaking.
That's true in any field, in any period.
The finding of the Higgs, for example, was published by two teams.
http://www.nature.com/news/nobel-laureate-challenges-psychol...
Title: "Nobel laureate challenges psychologists to clean up their act"
Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming."
I'm sure there is great value in putting it out there if people find it later. The problem is two fold: firstly where do you put it (journals with limited space and limited time for reviewing content aren't going to be all that interested, and people aren't going to trust some random page on the Internet), and secondly will people even look for it before starting their work?
This affects many scientific area and not just this one - just think how many instances there have been of medical experiments going ahead based on the results of a study or two despite the fact that a bunch of other studies contradicted the findings? Those studies either were not where they could be easily found, were not searched for, or were not published at all at the time. This is particularly problematic when there is a conflict of interests due to studies being commissioned by those selling the products/processes/ideas they hope the studies will support: there is a commercial drive to publish every success and down play every less positive result.
(Though, of course, most of those have been published in a reputable journal.)
Still, if the content was both checked for being reasonable and also classified well, then this type of online database could possibly provide a lot of value to researchers.
A lot. Failed experiments can be the basis for an improved one attempting to fix what made it go wrong the first time, even if "only" for a master thesis study. It can also be a real asset for anyone writing a paper on a similar subject with working results since they can cite the failed experiment to quickly justify why they didn't test some variations of their hypothesis etc. In a world where paper are often limited in number of pages, being available to list failed attempts with citations is awesome compared to having to write a small paragraph with approximative and most often bad explanations for each failures.
A first step might be to put such (non-)results in short papers and upload them to an open access repository in the kind of arXiv or HAL for instance.
What if I am a researcher who has a similar idea and is saved 6 months testing because of your work? Or what if I read it and see that some of it is valuable and look into a related area.
Of course, if there is a tremendous amount of research published on, say, how to make a perpetual motion device, I can see how that could waste people's time.
There is also a chicken-and-egg problem here with changing the culture. Sure, I could decide tomorrow that I will no longer battle for space in the high-impact journals that usually display this bias — in favour of more enlightened journals like PLoS ONE ... and I wouldn't suffer, I have a job... but what about my grad students and postdocs? The consequences for them would be real, because biases also still exist in the minds of professors, hiring committees, and even their peers. 10 papers in PLoS ONE just are not considered the same as 10 papers (or even 5 papers) in more "established" journals. There is thus a lot of implicit pressure to just abandon studies that show null results and pursue different ideas until you find an effect.
There is also huge pressure today (very explicit) to continuously generate "novel" ideas. If you want a competitive research grant now you have to generate a groundbreaking idea. If you want to publish in the top journals you have to generate groundbreaking ideas. Amazingly there are colleagues that have a success rate with grants and with high-impact publications so regular (EVERY SINGLE YEAR) ... that wow, how amazing it must be to be able to generate a brand new groundbreaking idea at such regular intervals!!! (sarcasm). And so there is yet another kind of bias present. (politics, the favoured few, the ivy-colorer-special-people, etc)
I could go on with other injustices but the bottom line is that [NEWSFLASH!] science is not objective, but like all aspects of life, is chocked full of biases, politics, special interests, and cow-towing to the bottom line (i.e. money).
s/science/the contemporary practice of science/
Science hasn't changed -- it means the same thing. Its practice has changed to the degree that we may be justified in calling some of it pseudoscience.
Those 'theories' are very limited in scope, ad-hock theories. Not theories in the meaning of "abstract or generalizing thinking" (Wikipedia).
Given the "publish or perish" thing in academia, and devaluation of negative results, it seems there's a strong incentive to obtain positive results. And this is likely to increase the incidence of researchers being pressured into tweaking data to obtain affirmative results.
Sure, null results are meaningless for complex hypotheses, but when testing a very narrow variable with a well-designed experiment, a null result is actually a very important result.
If that's true, if a hypothesis is vague enough to resist meaningful testing, then a positive result would be equally meaningless. But I see we are in agreement.
Nice thought, but there's already the JIR, the Journal of Irreproducibe Results: