Machine learning of neural representations of emotion identifies suicidal youth | Hacker News Reader

Machine learning of neural representations of emotion identifies suicidal youth

methodsman.com

103 pointsphr4ts74 comments

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts."

Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data set? There's no mention in the abstract of testing it against a fresh control set and that's not promising.

https://www.nature.com/articles/s41562-017-0234-y?error=cook...

scihub has it. looks like leave-one-out cross validation?

"A Gaussian Naive Bayes (GNB) classifier trained on the data of 33 out of 34 participants predicted the group membership of the remaining participant with a high accuracy of 0.91 (P<0.000001), correctly identifying 15 of the 17 suicidal participants and 16 of the 17 controls"

Correct. They also tested leaving out 9 of each group for cross-validation and got 76% accuracy in that case.

"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported."

Sounds like they overfit their cross validation score and reported that. The data is actually available here though:

http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/

How does LOOCV overfit the cross validation score?

The simplest answer is that every single data point was used in some part of the training, and then later used to give test validation.

We have no idea how this result would work on a new data point that has not been used in training.

It's bad statistics, bad data science.

Multi-fold cross validation is an established technique. At no point is any data that was used to train a model also used to validate it.

> One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model.

https://en.wikipedia.org/wiki/Cross-validation_(statistics)

You might object that it's difficult to achieve the level of inter-fold isolation required to make the technique sound, and indeed if you search the comments there's some question as to whether or not there was an information leak in their feature selection process. In that sense calling it "bad statistics, bad data science" might be reasonable, but it's also a powerful technique, so I don't think it's reasonable to dismiss out of hand without being more specific.

LOOCV can have high variance, but overfitting?

What's the smallest sample size anyone has used to get > 90% on the MNIST set?

I was thinking they might have "overfit" by picking hyper-parameters that happened by chance to result in a high CV score but wouldn't perform well in a holdout set. I could be wrong though, that's just my intuition.

17 subjects per group is extremely small.

Looking at it either from a machine learning or statistical point of view, using such a small sample is problematic.

This is the chronic issue with fMRI studies, since administering an fMRI is extremely expensive, and has led to some very difficult to reproduce results in the field.

I believe some pretty fundamental fMRI spatial autocorrelation functions have been called into question as well (1). Sounds a bit like PowerPoint's autocontent wizard.

(1) http://www.pnas.org/content/113/28/7900.abstract

Oh, but the paradigm-shifting revolution of a paper, sorry: "poster" surely is "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon": http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

People love the "n=XX is far too little data!" argument, yet it's more complicated than that. Sometimes 600,000 is too little, yet sometimes 17 is enough.

Example: you believe a newly found plant species is toxic. You give it to 17 "grad students volunteers", while giving a placebo to 17 others. All in the first group die aa gruesome death within 20 hours. None of the others do.

Result: yes significance. (also: tenure!)

I'm not saying that this study is significant (the statistics seem to be slightly beyond my event horizon), and your criticism also stops short of an outright dismissal of the research. But sample size alone makes for a bad measure of quality. Yes, even p-values are better.

I really hope you wouldn't get tenure for a study that killed all your subjects.

Subjects? But they're volunteer grad students!

Subjects, minions, whatever you want to call them :p

The technical term for that is "effect size".

I think that a small sample size is mostly an indicator that one needs to treat the results with far greater caution.

Effect size is very important in this. To continue your grad student murder example, it's completely trivial to determine which plant a student was given, based on whether they are dead or not. It becomes trickier if you measured something a bit less cut-and-dry, such as the incidence of headaches, or variance in a few voxels of a noisy MRI.

I am extremely stoked that a huge sample of 17 people have decided to upvote my comment ;-)

in the paper they state they performed " 34 “leave one participant out” cross-validation cycles (folds)", the dataset still seems to small to be able to draw any conclusion though.

Doesn't 91% seem far too low to be useful for the general population? Consider that only 7% of the background population experiences one or more depressive episode per year[0] (edit: okay maybe 8% in youth). Assuming independence and using the higher 8% background rate figure for youth, .91 * .08 = 7.3% of the population will receive a true positive result and (1-.91) * (1-.08) = 8.3% of the population will receive a false positive result. This is "pretty bad" — false positives outweigh the true positives — making the value of a positive result useless.

(Consider what happens to people so-diagnosed as suicidal when in fact they are not (false positives). Involuntary psychiatric imprisonment is a terrible thing if it isn't absolutely necessary.)

[0]: https://www.healthline.com/health/depression/facts-statistic...

> I don't have the stats grounding to come up with the proportion of true positives to false positives, but I suspect this would be "pretty bad" — vastly more false positives than true positives

IANAStatistician, but let’s consider the system is right 91% of the time and we try to detect those 7% you mentioned. Let’s take 1000 people. 70 people are depressive and 930 aren’t. Out of those, 700.91=63 will be correctly classified as depressive by the system and 9300.91=846 will be correctly classified as non-depressive.

That leaves us with 63 positives, 846 negatives, 7 false negatives and 84 false positives. False positives largely outnumber false negatives, but they also outnumber the true positives.

(if a statistician read this, please correct me if I’m wrong)

Yeah. I came to largely the same conclusion but used the 8% number for youth, given the current title.

Isn't the point of research to advance science one step at a time, not go from "does this look promising" to "yes, it works perfectly 100% of the time" in a single quantum leap.

Will it be that 10% of people are suicidal and it always predicts non-suicidal?

Will it be that accuracy actually means AUC?

Will it be that they are reporting predictive skill on the training data?

Not only that, the researchers admit that 80% of suicidal people deny being suicidal. Then, how can they be sure than the ones in the control group are not suicidal?

That wouldn't call the results into question. It would, in fact, strengthen them.

That's because the measured difference between the groups would be lower (because the real difference would be lower if the groups are more alike than you think).

Say you're testing a drug that's supposed to make people taller. You don't know it yet, but it really does make everyone grow 10cm overnight. You give it to half of your volunteers, and the other half gets placebo. The next day you find that the first group grew by 10cm compared to the control.

Now say your grad student messed up and half of the control group also got the real thing instead of placebo. Those also grew by 10cm, making the average in the control group 5cm, and your treatment group's effect is suddenly lower.

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance).

[...]

The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." https://www.nature.com/articles/s41562-017-0234-y

The winner is #3: data leakage leading them to use predictive skill on the training data.

If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.

>"If they included feature generation in the training process and ran it once per fold, it would be OK"

How so? The data used as validation in one fold would be used to determine features in the next...

Anyway that isn't what they did. From the supplements:

"To identify the most discriminating concepts, a reiterative procedure analogous to stepwise regression was performed. In the first iteration, the group classification was performed using only one concept at a time, determining which single concept of the 30 resulted in the highest classification accuracy. In the second iteration, the classification was performed using pairs of concepts, namely the single concept that produced the highest accuracy in the first iteration as well as each of the 29 other concepts. All pairs that produced at least as high an accuracy as achieved on the previous iteration, were explored in the third iteration, where triplets of concepts were used, namely the pairs that produced the highest accuracy in the previous iteration, plus each of the remaining 28 concepts. Such stepwise addition of discriminating concepts continued until adding any one of the remaining concepts resulted in a decrease in accuracy. An analogous procedure identified the most discriminating locations."

But I still think even in your case they are doing:

  train: abc; val: d -> score1/ features0 -> features1
  train: abd; val: c -> score2/ features1 -> features2
  ...etc

score2/features1 would all contain info from c, etc.

It's training data. There's 17 suicidal and 17 non-suicidal scans, for a total of 34 scans. They trained 34 models, leaving one scan out each time. Of those 34 models, 31 correctly predicted the left-out scan.

IANAStatistician, but this seems like a trash result.

How so? Isn't there a 50% chance of getting it right by pure chance, but they got it right 91% of the time instead?

Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.

How did they adjust the model/features based on CV performance? It looks to me like they did LOOCV.

Read the second paragraph I quoted above:

"The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)."

The features were chosen using the same data as used to assess predictive skill.

That quote does not support your summary, unless you are basing it on the information not explicitly mentioned. (I.e. they didn't say that they were only using training data to select features, but if they are any competent, they did.)

See the last part of this post: https://news.ycombinator.com/item?id=15598117

Can you provide pseudocode consistent with what they described (in the post you responding to) that wouldn't lead to leakage? I can't see it.

Select a training set, leaving out one sample for validation. For all features, train a classifier on the training set using that feature. Keep the one that gives the highest discrimination score on the training set. Repeat with more features. Then evaluate the final classifier on the validation sample, which has so far not been seen in any of the steps. The result provides an estimate of the risk on unseen data from the same distribution.

To get the estimation variance down, you can repeat this for all possible choices of validation sample. That means, you start the feature selection process on the new training set over from scratch and obtain another risk estimate. If they kept the features selected earlier, that estimate would be "contaminated" and not independent, but if they correctly start over, the procedure is valid.

My understanding is you are saying create N (N=34 in this case) different parallel models that use different features/etc. Then take the average (or whatever summary stat) of the accuracies to get the predictive skill.

When we want to use these models, we run new/test data through all N=34 models in parallel and calculate a prediction from each. Then somehow these predictions need to be combined (one again an average, etc). This is the average of the predictions, not accuracies/whatever.

Where was the step combining these predictions present during the training? It seems your scheme necessarily calculates an accuracy based on a different process than needs to be applied to new data.

No, when you want to classify a new sample, you take a model trained on the complete labeled data you have and use the prediction of that. The validation procedure using those 34 models trained on subsets of the data is just to tell you how accurate you should expect the result to be. Afterwards, you can throw those models away.

Of course you could build an ensemble model, but if you want to know the expected accuracy of doing that, you need to include the ensemble-building into your validation procedure. (Or use some theorem that lets you estimate the ensemble performance from that of individual models, if that is possible.)

>"when you want to classify a new sample, you take a model trained on the complete labeled data you have and use the prediction of that."

Using which set of features? You have 34 different models with different features...

You run the whole training process on the complete data. Including feature selection.

I see. So usually what you would do is run the CV a bunch of times to test various features/hyperparameters, knowing this will overfit to the data used for the cv.

After deciding on features/hyperparameters (based on the overfit cv), you train the model on all the data used for cv at once. Then test the resulting model on a holdout set (that was not used for the cv). The accuracy on that holdout would then be the accuracy to report.

This sounds much like what you are describing, except you only do one cv and do not use it to decide anything. The cv is only to give an estimate of accuracy.

Is that correct? It does seem to legitimately avoid leakage. However, it seems impossible that an anything close to optimal feature generation process or the hyperparameters were known beforehand. Do you just use defaults here?

In this case nested cross-validation would have been the proper way to do this. Run your entire model selection process (scaling - feature selection w/ CV - model selection - hyper paramter tuning w/ CV) on each of the folds in the outter CV loop. That will tell you how good your process is at building a model that generalizes.

"Words like death and cruelty differentially activated the left superior medial frontal area and the medial frontal/anterior cingulate in the individuals with suicidal ideation – these are areas associated with self-referential thought." I wonder how they reacted to "alive" and "humane"

It's the kind of "studies" you call BS on first, then go on to figure out the details. Not a very scientific process for sure, but always produces the correct result.

https://www.naturalblaze.com/2017/03/scandal-mri-brain-imagi...

At least post the original article[0] if not the paper[1], rather than some weird alt-health website.

[0] https://www.sciencealert.com/a-bug-in-fmri-software-could-in... [1] http://www.pnas.org/content/113/28/7900.abstract

Slightly off topic but the book "Change your brain change your life" was pretty interesting. Perhaps not as scientific as some would prefer, but none the less thought provoking.

Basically you extract a matrix representation of the active or inactive regions that is classified and have DNN learn it like you would learn images, is that a correct assumption?

To the moderators, the title would be more accurate with 'fMRI' as opposed to 'MRI'. The latter is typically used to examine structural brain elements, whereas fMRI is thought to correlate with brain activity and, by extension, thought.

Confusing the two would lead to the more unusual conclusion that suicidal ideation is associated with abnormal brain connectivity, while the authors are instead focusing on neuronal activity.

Specifically fMRI measures blood flow across the brain (the BOLD response) which is correlated with neuron activity. It has good spatial resolution but poor temporal resolution [0], compared to EEG which gives you good temporal resolution but poor spatial resolution.

[0] i.e. you know with precision where in the brain activity occurred, but less precisely when it occurred in time

I didn't even really realize I had these confused until you pointed it out. This makes a lot more sense and helps me understand the results. At first I was confused at how brain structure analysis predicted suicidal tendencies/thoughts.

So what will they do after they detect you are suicidal? Stick you in a psych ward? Yet more attempts at taking away the rights of those going through trauma.

That seems like putting the cart before the horse.

Diagnosis tools could mean faster access to treatment. Currently in the UK the waiting list for access to mental health treatment is on the range of two to three years. Transforming "suicidal ideation" from a "vague human-given diagnosis" to "tool-given diagnosis" makes it politically easier to push for that.

In any case, that's not going to happen based off a single study with 91% accuracy.

Maybe they should use this test before gun purchases... I don't think someone suicidal should purchase a gun...hell I don't care if they kill themselves, but lately a lot of suicides were mass suicides...we don't need more of that shit.

Problem is people buy guns when they aren't suicidal.

Tip: If you own a gun and are feeling suicidal, give it to a trusted person for safekeeping.

It can't hurt to get rid of the gun if you're suicidal. But would it actually make a difference? Suicide rates across countries aren't related to gun availability.

I think false positives can hurt.

Got any citation for that? Sounds like the kind of thing someone would confidently assert on principle.

Suicide is definitely linked to gun availability.

"A study by the Harvard School of Public Health of all 50 U.S. states reveals a powerful link between rates of firearm ownership and suicides. Based on a survey of American households conducted in 2002, HSPH Assistant Professor of Health Policy and Management Matthew Miller, Research Associate Deborah Azrael, and colleagues at the School’s Injury Control Research Center (ICRC), found that in states where guns were prevalent—as in Wyoming, where 63 percent of households reported owning guns—rates of suicide were higher. The inverse was also true: where gun ownership was less common, suicide rates were also lower."

https://www.hsph.harvard.edu/news/magazine/guns-and-suicide/

Gun ownership in the U.S. is strongly correlated with socio-economic status, locality, etc.

Everybody points to the Australian example, where suicides declined after the 1996 gun control legislation. But unemployment in Australia peaked in 1995 and declined precipitously afterward until 2009.

Given everything we know about suicide rates in other countries, and about changes in suicide rates domestically (e.g. recent increase as gun ownership goes _down_[1]), it would very odd if gun ownership was a root cause of suicide.

That said, in a country with a strong gun culture like the U.S., I would totally expect a generational dip in suicides if we substantially removed access to guns. But then I'd expect it to normalize when suicidal individuals became more comfortable with other methods. Just like with mass shootings, there's a strong imitation effect. Take away the model that people imitate and it might be awhile until there's a regression to the mean.

Even so, that's still reasonable justification for limiting access to guns--saving tens of thousands of individuals. I'm not sure I'd agree with such a policy prescription because of the insane gun politics, but it's quite defensible from a public health perspective.

[1] Number of guns have increased but they're concentrated in fewer households.

https://www.hsph.harvard.edu/means-matter/

Tl;dr?

* Many suicide attempts occur with little planning during a short-term crisis.

* Intent isn’t all that determines whether an attempter lives or dies; means also matter.

* 90% of attempters who survive do NOT go on to die by suicide later.

* Access to firearms is a risk factor for suicide.

* Firearms used in youth suicide usually belong to a parent.

* Reducing access to lethal means saves lives.

Those are all just rationalizations. They can't reflect anything intrinsic about suicide risk if they can't predict relative suicide rates outside the U.S.

Suicide is an epiphenomenon of larger socio-economic issues. Among OECD countries the U.S. comes in the middle of the pack. If guns were a causative factor, then considering how prevalent they are (by a ridiculous factor!) our rates should be much higher:

https://en.wikipedia.org/wiki/Suicide_in_the_United_States#/... http://foreignpolicy.com/2013/05/03/how-does-americas-suicid... http://www.oecd-ilibrary.org/sites/health_glance-2011-en/01/...

Any correlation between guns and suicide in the U.S. is easily understood in terms of modeling--guns are how Americans kill themselves. Take away the guns and, yes, there'll be a dip in suicide rates, until Americans learn how people kill themselves elsewhere around the world. Heck, they're already learning that with opioids.

Let's go back to what I said about Australia. I claimed that the change in Australian suicide rates is better understood in terms of the unemployment rate. Now let's test that hypothesis. [... google google google ... ] Here we go:

  Suicide has reached a 10-year high in Australia as 3027
  people killed themselves last year, the largest cause of
  death among 15 to 44-year-olds.

  Last year, 12.6 people in every 100,000 killed themselves
  compared to 12 the year before, 11.4 in 2012 and a low of
  10.4 in 2006.

  -- http://www.theaustralian.com.au/news/nation/suicide-rate-in-australia-reaches-10year-high/news-story/cb5d8384aadb571778775bda236f3c35

What was the rate in 1995, the year before the gun control law? 13.0. (https://www.aph.gov.au/About_Parliament/Parliamentary_Depart...)

So suicides were at their lowest the very same year that unemployment was at its lowest (2006)? Check! And they rose as unemployment rose? Check! To the point where they're back at the pre-law level? Check!

I won't deny that there's some nuance here that we can tease out, but if you read the actual papers that link guns to suicide, they do a much worse job at nuance. In fact, not a single one of the papers I've read even considered the unemployment rate. Which is patently bad science.

Correlation is not causation. All the gun suicide papers do is point out specious correlations. But you don't need a degree in statistics to know this. And you don't need a science degree to be able to see the gargantuan holes in these arguments--that the correlations have simpler explanations.

I'm not denying that gun control could appreciably effect suicide rates. Imitation and modeling have huge effects--much more well-established than the supposed gun effect. So huge that even news media abstain from reporting suicides, especially methods of suicide. Saving thousands of lives with gun control, even if it's an ephemeral gain, is an absolute benefit that's worth debating about. But let's just be honest about this stuff.

I hope that you submit your thoughts to a place more formal than this comment, I find the unemployment thing interesting. Particularly in Australia with $540 per fortnight unemployment and free healthcare, you could compare the difference between suicides and safety nets, and the overall suicide cost of poor safety nets.

It does seem like you agree in your last paragraph though though... gun control would lower suicide rates. People would try other methods which have higher failure rates, people's lives would be saved. Not all of them, but an appreciable amount.

https://en.m.wikipedia.org/wiki/List_of_countries_by_suicide...

Lots of Europe has a higher suicide rate than the US. Within the US, as the other response said, gun ownership is confounded by all sorts of other factors.

"Lots" is strong, given that the only western European countries higher were Finland and Sweden I could see, which is known to be caused by the long and dark winters.

Intentionally denying 1 in 10 healthy adults their rights is illegal, no matter how much good you're doing by denying some smaller group their rights.

(It's also not clear that even if they have suicidal ideation that they're not entitled to gun rights, but I don't feel like having that debate today.)

Not having ready means to off yourself when you are suicidal can't hurt?

When Australia made it harder to get a gun, their deaths by suicide dropped significantly, even moreso than gun homicides in fact.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2704353/

The Australian gun buyback recorded only a ~30% replacement rate for suicide by guns. i.e. only 30% of predicted gun suicides were achieved through another method.

Unless the 'trusted person' is an immediate family member this is illegal in california as of last year http://leginfo.legislature.ca.gov/faces/displayCodeAndBill.x...

I don't know about other states but it's likely any that require background checks for private transfers ("closing the gunshow loophole") would make this behavior illegal.

The guidelines ask us to please not do this kind of thing:

> Avoid unrelated controversies and generic tangents.

https://news.ycombinator.com/newsguidelines.html

Reply on news.ycombinator.com