Laypeople can predict which social-science studies will replicate successfully
journals.sagepub.com
journals.sagepub.com
"The materials for each study included a short description of the research question, its operationalization, and the key finding. These descriptions were inspired by those provided in the SSRP and ML2, but were rephrased to be comprehensible by laypeople. In the description-only condition, solely these descriptive texts were provided; in the description-plus-evidence condition, the Bayes factor and its verbal interpretation (e.g., “moderate evidence”) were added to the description of each study."
I was surprised to read about the low reproducibility rate, how can you call it social "science" if you can't reproduce it?
NB: this is not specific to the social sciences. psychology, cognitive sciences, medicine(!), and other fields have a 'replication crisis' also. there are some really good papers on this. i am not aware of systematic queries into CS/ML/DL papers, but from my own experience: when you pick up a paper that claims 9x% accuracy, the chance that you are able to replicate that number is in line with replication % in other fields.
From all my friends who pursued STEM, I assume this is happening because of the pressure on people pursuing higher education at all costs, not finding enough qualified jobs, getting stuck in academia and printing low quality papers to get a promotion / keep getting another public grant.
If you add in to the mix results influenced by the current political climate, what your peers think, what your sponsors want to prove for financial gain, the situation gets bleak very fast.
Irreproducible papers are the abandoned OSS projects of software engineers in private tech + corruption + public money, a recipe for disaster.
OSS is often someone learning, or who has an itch to scratch, and put some code out there that may or may not solve your problem or work right. The "lack of warranty" clause in most open-source licensing is really important here, because it's largely designed to say "hey, I did a thing but make no guarantees so use at your own risk", whereas a published paper says "hey, a number of experts and people who should know all agree that this is a thoroughly-researched and well-thought-out position, and you can probably consider it to be true (or nearly so) and base some of your decisions on it." I think that change in context is really important to consider when making the analogy here.
This is a myth. Modern "peer review" simply means checking to see if the claim is interesting and if the proper Word or LaTeX template was used. Peer review meant replication in the distant past before the modern Grants and Impact Factor system was built.
I got peer review from two reviewers on a paper submitted to Frontiers last week--it consisted of much more than that. Sure, the reviewers couldn't replicate our exact work themselves, but the paper's methodology was also heavily considered, and both reviewers asked for modifications to be made on different parts of the paper.
It's an imperfect system, but it's not as if there's a rubber stamp floating around. The goals of the the peer review system vs. open source are fundamentally different, and the way the products for both are treated should reflect that.
And definitely there is some form of age group bias, if the large majority were students (first year or otherwise) the sample must be dominated by 20-25 years people.
Maybe (or maybe not) a sample with more aged people (i.e. with more life experience) would have reached more than 59%.
After taking that the result in this paper doesn't surprise me. I'm educated, but not in Psychology, and I think most people can tell which studies are more likely to fail to replicate.
As an example:
"If subjects hold a heavier clipboard they feel more authoritative" - that's a pretty unexpected and surprising result. I expect it won't replicate.
"People will give more to a charity if the overheard for that charity is already covered." That feels pretty common sense to me. If you're donating, you probably want your money to go to the cause, and not the overhead of the organization gathering donations etc.
I remember working for a manufacturing company where I drove a forklift (during the summer).
One of my most vivid memories is walking into the breakroom and seeing on the television a scrolling banner that declared homosexual couples were less likely to have children than heterosexual couples.
And I remember everyone in that room laughing their asses off. And it's not as if no one in that room realized the reason for the study was so they could take into account things like adoption, etc. It's that the result was so obvious, even taking that into account, that it was amazing that someone was PAID to come to a conclusion that everyone knew without the money.
And this is the crux of the problem with "science". It wants to be "interesting", so it will literally try to drum up something against what "everyone knows".
So the idea that what everyone over generations "knows" is generally more applicable than "science" is not surprising at all.
recursion :D
I won't quote the science, I'll simply say that things are knowable outside of a system...
The truth is that 99% of the population believes that homosexual couples will have less children then heterosexual...
The surprising result for the uneducated? That homosexual couples have any children...
They're using the percentage in the original comment/study and transferring it to the recent example.
That doesn't mean that 41% would not be convinced it could be replicated though since it's likely example related.
I did a quick search and found a study about percentages of couples raising children [1]. They have a table by couple type and marital status. The difference isn't as stark as you might think.
Looking at the smallest difference in the table, married female/female partnerships have a 30.2% chance of currently raising children, while married male/female partnerships have a 38.7% chance. Is it obvious to you that this would be so close? Would you be surprised if there were countries where the numbers are reversed?
[1] https://williamsinstitute.law.ucla.edu/publications/same-sex...
It is obviously important for population and demographic research to know just how age, type of couple, divorce, income, etc. effects family size with actual estimates.
Your post just did a study on that topic. Should you get funded for that?
A personal favourite example is that meme darling, the Dunning-Kruger effect. In addition to often being an extremely simplified version of the original proposition, which is a lot more subtle than "dumb people think they're smart", it's often presented as if (social!) science has uncovered a universal trait in the human race. But really this effect is based on very few studies, and they were mostly done on American college students. I wonder how well it replicates...
Students are expected to participate in a few studies now and then as they will be on the other side at some point.
But the newest bad one was 2012 and you could say most of the bad ones had a socialist lean.
And the 67% when the people is informed about the strength of evidence is even less impressive. How much accuracy would have a parrot that just repeat the information?
To be fair they translate the strength to the evidence from a numeric scale to a simple words scale, and the participants should translate it back to a numeric scale. The real question is how good is people doing this task with random numbers, without additional information like the description of the study.
Edit: but seriously, I would assume that this applies to hard sciences as well, if you replace “laypeople” with “domain experts”. Plausible hypotheses are cheap; designing a study and collecting data to prove them is hard. Many studies confirm something that feels obvious to researchers in the field, but being able to write that thing down as fact with a citation allows people to move a step forward. I would say it’s the extreme outlier study where an implausible hypothesis is proven true.
Surely it’s the expected outcome.
The opposite would be something like ‘most social science results are counterintuitive’.
So much of our political rhetoric is just basic one dimensional math. "have and have nots", "wage gap", and so on.
The World would be a better place if more people understood exponential growth, statistics, game theory, and so on. It hampers our politics, when the rhetoric needs to be reduced into basic algebra.
"In a sense we've come to our nation's capital to cash a check. When the architects of our republic wrote the magnificent words of the Constitution and the Declaration of Independence, they were making a costly signal of commitment. But it's obvious today that America has failed to support the separating equilibrium... We have also come to this hallowed spot to remind America of the fierce urgency of hyperbolic discounting.... I have a dream that my four little children will one day live in a nation where their skin colour will not be a sufficient statistic of the content of their character!"
https://www.quora.com/Are-the-social-sciences-merely-dressed...