At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.
>At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.
Absolutely.
If you run an experiment a day and get p < 10^-9, your priors, your multiple hypothesis correction, even your interpretation of p-values approximately don’t matter. Running social sciences experiments with p < 0.05 threshold is where things get weird.
>even your interpretation of p-values approximately don’t matter
"Small number means good" is not a sufficient working understanding of p-values for doing science.
That is literally mindless statistics. Which coincidentally is the name of the article I talked about. Did you read it?
(In a social science, if your p-value is 1E-5 or something, the most likely interpretation is that you are doing something very wrong)
another good article on misinterpretation of p-values and confidence intervals is: Greenland, S., Senn, S.J., Rothman, K.J. et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31, 337–350 (2016). https://doi.org/10.1007/s10654-016-0149-3
A decent compromise would be to at least require meta-data to sufficiently exclude some flaws. A different approach could be to have researchers document and publish the process of th research, similar to a git-repo with the main branch being completely off limits to history-rewriting.