From the paper: "Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%." (http://www.pnas.org/content/early/2016/06/27/1602413113.full)
Choosing a bad algorithm for your software is as much of a bug as a null-pointer crash, and is something that needs to be tested for when building software.
"In most psychological research, improved power of a statistical design leads to a prior probability approaching 1/2 of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by “success” is very weak, and becomes weaker with increased precision. “Statistical significance” plays a logical role in psychology precisely the reverse of its role in physics." http://cerco.ups-tlse.fr/pdf0609/Meehl_1967.pdf
In other words, the fMRI researchers have been testing a null hypothesis known to be false beforehand, so statistical significance tells us nothing of value, and definitely nothing about the theory of interest.
This paper is really interesting, because the authors don't seem to realize that this is what they are discovering. Getting a "statistically significant" result is the correct answer for the tests to return, these are not false positives. The problem is a poorly chosen null model.
Specifically, they started with some resting state data (a condition where the subjects just lie there, attempting to remain alive). They then took this data and imposed fake task structures over it, as if the subject were doing two different things (e.g., condition #1 starts at t=0, t=1 min, t=3 min, t=6 min; condition #2 starts at t=2, t=4, t=5 min). Once this is done, the task+data is submitted to their analysis pipeline.
The null hypothesis (fake condition #1 == fake condition #2) has to be true, by construction. This assumes that the original data doesn't have any structure, which may or may not be true (they discuss it a bit towards the end of the paper).
The null hypothesis is not only that condition 1 == condition 2, it involves other assumptions being made. From their own description, they knew it was false before even doing this analysis and used this knowledge to design the study. Apparently there is some incorrect assumption about autocorrelation being made.
It seems the authors did not really grasp what hypothesis they were testing, leading to their confused (but still productive) description of the problem. I go into more in this post: https://news.ycombinator.com/item?id=12032772
On the other hand, no researcher would knowingly publish results affected by software bugs, e.g. null pointers or arithmetic errors... hopefully.
"... a 15-year-old bug was found in 3dClustSim while testing the three software packages (the bug was fixed by the AFNI group as of May 2015, during preparation of this manuscript). The bug essentially reduced the size of the image searched for clusters, underestimating the severity of the multiplicity correction and overestimating significance ..."