Hacker News Reader

Comment by nieve | Hacker News Reader

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts."

Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data set? There's no mention in the abstract of testing it against a fresh control set and that's not promising.

https://www.nature.com/articles/s41562-017-0234-y?error=cook...

View on HN

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported."

Sounds like they overfit their cross validation score and reported that. The data is actually available here though:

http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

LOOCV can have high variance, but overfitting?

What's the smallest sample size anyone has used to get > 90% on the MNIST set?

I was thinking they might have "overfit" by picking hyper-parameters that happened by chance to result in a high CV score but wouldn't perform well in a holdout set. I could be wrong though, that's just my intuition.

How does LOOCV overfit the cross validation score?

The simplest answer is that every single data point was used in some part of the training, and then later used to give test validation.

We have no idea how this result would work on a new data point that has not been used in training.

It's bad statistics, bad data science.

Multi-fold cross validation is an established technique. At no point is any data that was used to train a model also used to validate it.

> One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model.

https://en.wikipedia.org/wiki/Cross-validation_(statistics)

You might object that it's difficult to achieve the level of inter-fold isolation required to make the technique sound, and indeed if you search the comments there's some question as to whether or not there was an information leak in their feature selection process. In that sense calling it "bad statistics, bad data science" might be reasonable, but it's also a powerful technique, so I don't think it's reasonable to dismiss out of hand without being more specific.

17 subjects per group is extremely small.

Looking at it either from a machine learning or statistical point of view, using such a small sample is problematic.

This is the chronic issue with fMRI studies, since administering an fMRI is extremely expensive, and has led to some very difficult to reproduce results in the field.

People love the "n=XX is far too little data!" argument, yet it's more complicated than that. Sometimes 600,000 is too little, yet sometimes 17 is enough.

Example: you believe a newly found plant species is toxic. You give it to 17 "grad students volunteers", while giving a placebo to 17 others. All in the first group die aa gruesome death within 20 hours. None of the others do.

Result: yes significance. (also: tenure!)

I'm not saying that this study is significant (the statistics seem to be slightly beyond my event horizon), and your criticism also stops short of an outright dismissal of the research. But sample size alone makes for a bad measure of quality. Yes, even p-values are better.

I think that a small sample size is mostly an indicator that one needs to treat the results with far greater caution.

Effect size is very important in this. To continue your grad student murder example, it's completely trivial to determine which plant a student was given, based on whether they are dead or not. It becomes trickier if you measured something a bit less cut-and-dry, such as the incidence of headaches, or variance in a few voxels of a noisy MRI.

I really hope you wouldn't get tenure for a study that killed all your subjects.

Subjects? But they're volunteer grad students!

Subjects, minions, whatever you want to call them :p

The technical term for that is "effect size".

I believe some pretty fundamental fMRI spatial autocorrelation functions have been called into question as well (1). Sounds a bit like PowerPoint's autocontent wizard.

(1) http://www.pnas.org/content/113/28/7900.abstract

Oh, but the paradigm-shifting revolution of a paper, sorry: "poster" surely is "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon": http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

I am extremely stoked that a huge sample of 17 people have decided to upvote my comment ;-)

scihub has it. looks like leave-one-out cross validation?

"A Gaussian Naive Bayes (GNB) classifier trained on the data of 33 out of 34 participants predicted the group membership of the remaining participant with a high accuracy of 0.91 (P<0.000001), correctly identifying 15 of the 17 suicidal participants and 16 of the 17 controls"

Correct. They also tested leaving out 9 of each group for cross-validation and got 76% accuracy in that case.

in the paper they state they performed " 34 “leave one participant out” cross-validation cycles (folds)", the dataset still seems to small to be able to draw any conclusion though.

Reply on news.ycombinator.com