Am I reading this right?
They are saying there is no reproducibility problem, because it's perfectly normal to be able reproduce just 39% of studies?
They are saying there is no reproducibility problem, because it's perfectly normal to be able reproduce just 39% of studies?
Whatever the actual error rates turn out to be, something like that is going to cut pretty deeply into measured reproducibility rates, isn't it?
On the other hand, if probabilities of methodological flaws are low for reproductions, then the multiplicative effects of the original method and the reproduction method are dominated by the original method, in such a way that more replications means multiplying probabilities together on the replication side, which (if flaw rates are less than 50%) goes to zero pretty quickly.
Meaning that consistency among reproductions is strong evidence regarding the validity of the original, and so the replications are worth the cost.
There's a significant qualitative difference between cases where replications are likely to be flawed vs. cases where they aren't. Cases where different replications are likely to have correlated flaws (even when flaws overall are unlikely) would also be problematic, because different replications might not give you additional information.
I think it matters to be very clear about all this, and not give any impressions that could be easily misconstrued to cast doubt on the utility of replications. Making some back of the envelope estimates using a 50% flaw rate strikes me as a dangerous way to talk about it (unless we have evidence that the flaw rate really is that high).
I have exactly zero idea what the actual ratios are. But the actual ratios absolutely do not matter for the purposes of illustrating how the methodological errors compound.
If the flaw rate is >= 50% and the number of replications is low, it presents a case where it's of little social value to do replications, depending on their cost (which is usually high).
From your original comment, the way a naive reader will view it is that replications often don't work, because you primed them with a number you just pulled out of nowhere (50%). That will matter much more than the other detail about it being multiplicative, which doesn't really matter much at all no matter what the flaw rates are, because it all hinges on the flaw rates of the replications and how many replications you do anyway.
I know that you did not mean to misrepresent it with the 50% number. I'm just harping on this because by happening to use a number like 50% it has the potential to distract and do more harm than good, regardless of anything else in your comment or even whether or not the 50% had anything to do with your main point.
Try building a bridge with that.
Social Science methodology is piss poor and needs to improve. People don't take it seriously for a reason.
Sounds like a situation that could lead to a millennium of stagnation, similar to geocentrism.