Brain-Imaging Studies Hampered by Small Data Sets

Here's a link to the associated paper just out in Nature:

https://www.nature.com/articles/s41586-022-04492-9

Marek et al. Reproducible brain-wide association studies require thousands of individuals (2022)

"Magnetic resonance imaging (MRI) has transformed our understanding of the human brain through well-replicated mapping of abilities to specific structures (for example, lesion studies) and functions1,2,3 (for example, task functional MRI (fMRI)). Mental health research and care have yet to realize similar advances from MRI. A primary challenge has been replicating associations between inter-individual differences in brain structure or function and complex cognitive or mental health phenotypes (brain-wide association studies (BWAS)). Such BWAS have typically relied on sample sizes appropriate for classical brain mapping4 (the median neuroimaging study sample size is about 25), but potentially too small for capturing reproducible brain–behavioural phenotype associations5,6. Here we used three of the largest neuroimaging datasets currently available—with a total sample size of around 50,000 individuals—to quantify BWAS effect sizes and reproducibility as a function of sample size. BWAS associations were smaller than previously thought, resulting in statistically underpowered studies, inflated effect sizes and replication failures at typical sample sizes. As sample sizes grew into the thousands, replication rates began to improve and effect size inflation decreased. More robust BWAS effects were detected for functional MRI (versus structural), cognitive tests (versus mental health questionnaires) and multivariate methods (versus univariate). Smaller than expected brain–phenotype associations and variability across population subsamples can explain widespread BWAS replication failures. In contrast to non-BWAS approaches with larger effects (for example, lesions, interventions and within-person), BWAS reproducibility requires samples with thousands of individuals."

This is consistent with a number of other studies in the past coming at the problem from different angles. This is a pretty well-done paper though, something I wish I had thought to do myself.

Lots to say about this. There's a great deal of rot, corruption, and mistreatment, with no consequences or recompense, between the lines of this paper.

That's the worst part of this for me, that this will get spun as a simple mistake people were making, rather than a broken academic system that people were taking advantage of, with no change or consequences.

My first programming gig was as a student assistant in college where I was writing Matlab for psychiatrists and neuroscientists. The studies they ran rarely had more than 30 people, and their subjects usually ended up in multiple surveys because the cost of performing an fMRI was super high.

They also had a subject recruiting problem. They were a children's hospital and most of their subjects were recruited through the email blasts they would send out to employees which meant most of the kids were upper-middle class.