Many Researchers Using Same Data and Hypothesis Reveals Universe of Uncertainty
osf.io
osf.io
That's... surprising, but keep in mind that we're talking about "soft" social sciences here.
Measuring soft quantities like "public support for government provision of social policies in response to increased immigration" is very different from measuring hard quantities like, say, the location of a rocket in space at a point in time using earth as a frame of reference.
I'm looking forward to seeing what Andrew Gelman and his colleagues will say about it at https://statmodeling.stat.columbia.edu/ -- surely they will want to say something about it!
They're saying that scientists, given the same data, will achieve widely different results, and that they can't even work out how they're getting these different results. That's surprising and concerning.
> This reveals a universe of uncertainty that is otherwise hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a new explanation for why many scientific hypotheses remain contested.
On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science.
If you dig far enough down the rabbithole, you eventually find that the numbers are being pulled out of someone's proverbial ass. But pulling numbers out of your ass and using them to make a conclusion is a lot better than pulling a conclusion out of your ass and using it to make numbers.
Hmmm, I think the variance in results is maybe a bit more of an issue in the softer sciences and the conclusions would be better worded to reflect that considering the context here, rather than implying they hold to a similar extent over science generally.
There is no doubt human failings also challenge physics etc (see previous threads here) and humility and clarity is appropriate there also, but the significant problems highlighted in this study seem more particular to areas of research that have long been regarded as less sure due to the nature of their domain.
> they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public.
I interpret it to mean that different researchers applied same math to same data and got different results.
After all, the science is as hard as it can be, it's the object of investigation that's fuzzy and shifting and actively refusing to be measured ;)
Granted, real science has error bars and there's no upper limit to how big those could get in theory, but in practice there's an upper limit to where a scientific approach is better than just crafting a compelling narrative.
This could infect literally any human endeavor that generates measurements, hard sciences included.
Or, as another poster observes, this result itself is affected by high variation and we end up in a Godel-ian loop of "this study is false" results.
Recently I thought about trying to combat this effect myself by using multiple different analysis methods. For example, in my case I'm thinking about trying multiple different regression approaches. My research is in the physical sciences, so I don't think in this case the overall conclusions will be that different because the general trend is clear from a scatter plot (often not true in social science...), but it'll still be an interesting exercise.
You used a lot of words despite not saying anything.
> And more type-II errors will be made. A six sigma result means there is a one in a million chance of getting the result we observed from our experiment if our maths-based theory is correct.
It means nothing of the sort. It means that if the theoretical explanation is wrong there is only a 1 in a million chance that the pattern seen in the data is due to random chance or noise.
As for you not understanding the first bit, that might be a personal limitation, or some sort of aphasia.
Setting the bar at 7 sigma means we will only reject the null (our math based theory) if the observed experimental data would have only occurred by chance, under the null, with a probability of less than one in ten million.
Hence, setting such a high bar to reject the math based theory means we are relatively biased towards preserving our mathematical preconception about the world rather than accepting (unexpected) physical evidence. If we set the bar at say, three sigma, the bias shifts towards accepting the physical evidence.
The point is, the p alpha criterion is not what makes an experiment rigorous.
You mean the only causal and verifiable and by far the most useful view about the world?
Do you think the microchip in your Phone that you post from was built off intuition and gut instinct?
>rather than accepting (unexpected) physical evidence. If we set the bar at say, three sigma, the bias shifts towards accepting the physical evidence.
So the petabytes of experimental data from the LHC don't qualify as physical evidence in your bizarro world. Got it.
This reads like the take that an overly impressed with themself 1st year Psych major would have after taking an experimental methods course.
Here it is again, more generically: as the p-alpha level for rejecting the null hypothesis decreases, the tendency for Type-2 errors increases.
Do you disagree?
In a world where time and resources are not a factor, an infinite amount of data could be collected to ensure both types of error are minimized. In such a world, alpha can be made arbitrarily small. In worlds that have limited resources to dedicate to experiments, a judgement needs to be made as to whether we are more comfortable with Type-1 or Type-2 errors. If for a given experiment it is decided to set an extremely low alpha (e.g. 7-sigma) it is being implied that we would rather preserve the prevailing theory (null) than reject it in favor of the experimental data.
If a social science experiment was funded at the level of CERN/LHB I have no doubt they would choose a more restrictive alpha, and likewise if LHB had less resources they would likely drop their alpha accordingly based on some power analysis. I say this as someone with a phd in a hard science. Obviously it doesn't excuse the field completely for their reproducibility issues. They should make a more concerted effort with their limited funds, perhaps using the model of particle physicists who pool funds to make a fewer number of experiments more rigorous through higher amount of data collection. But this is all aside the small point I originally intended to make..
While the real sciences, don't have it: physics, chemistry, biology, etc.
Incidentally, I'm writing this in a place called "Physical Sciences Building".
I would only classify math and some physics as hard science.