Algorithm accurately predicts when teens have suicidal thoughts, behavior | Hacker News Reader

Algorithm accurately predicts when teens have suicidal thoughts, behavior

news.byu.edu

37 pointssizzle46 comments

Very sloppy statistics. A lot of explanatory analysis on the model is done but the model is hardly justified in being predictive in the first place.

The paper (https://journals.plos.org/plosone/article?id=10.1371/journal...) [does not mention the sensitivity or specificity of the model at all, only mentions a '91% accuracy' number]* on a biased dataset (where the number of suicidal cases is oversampled and non-suicidal cases are undersampled), without even mentioning exactly how much they over/undersampled.

* I missed the ROC curve on page 7. However it's not clear if this ROC curve was computed on the under/oversampled dataset or the original.

> The paper [url snipped] does not mention the sensitivity or specificity of the model at all

There's a full ROC curve in Figure 2. Just eyeballing it, it looks like they get both sensitivity and specificity in excess of .9 in the top left corner (I didn't try to measure it precisely).

It would certainly be helpful to have more information about the over/undersampling.

I did miss that, oops. That said, it's again unclear if that curve is pre- or post- biased sampling. If it is on the original data it is quite good, but with a heavy biased sample fairly meaningless.

If it's not made clear, post-sampling analysis is the safe assumption.

"You have to oversample your minority class so your data is balanced" is an urban legend that needs to die. It is never necessary, unless you are using extremely outdated model fitting methods, and even then, it would only be needed in training. It is completely unnecessary in evaluation, and metrics on biased data are going to be biased (obviously!).

If I hear this in an interview, I'm going to assume you do data science by blindly copying random blog posts.

While I agree with your overall sentiment, it is important to understand the relationship between class (im)balance and ROC curve performance. A very short article which does a great job explaining this is [0]. There are, of course, other performance metrics that are appropriate in the presence of class imbalance, such as precision-recall curves, so I wouldn't go so far as to say "metrics on biased data are going to be biased". Some metrics can correct for the class imbalance bias. Others can't.

[0] https://www.researchgate.net/profile/Jake-Lever/publication/...

What do you mean by class imbalance bias?

Agreed. ROC AUC is fairly robust to over or undersamping a class in determining if your classifier is predictive.

See my sibling comment. Suppose you have data with 0.95 positive class and 0.05 negative class. You can achieve high ROC AUC with the classifier that blindly predicts the positive class. It may be "predictive" (after all 0.95 of the data is positive), but I would hesitate to praise such a classifier.

Even if this works, the only question it answers is which questions on the survey correlate with considering suicide. The title is deceptive in that regard.

Common reasons for suicide the model found:

- Being threatened or harassed through digital media

- Being picked on or bullied by a student at school

- Exposure/involvement in serious arguments and yelling at home

Waow guys the computer solved the problem! Or not, everybody knew this already. They were "surprised" that family arguments have an effect? Anybody who knew kids with divorced parents growing up already know this.

Oh and notice it says "suicidal thoughts and behavior", it doesnt differentiate. Girls are more likely to have thoughts but less likely to act, pretty well known, so maybe it doesn't make sense to treat thoughts as seriously for them but this model makes it the same.

Oh and look at the survey they used, the CTC thing: https://web.archive.org/web/20100527132022/http://download.n...

Its asking obvious quesitons like "are you taking lots of drugs", "how is your relationship with family", "how engaged are you with peers and community socially", that kind of stuff. Like legit half of it is drug use questions. Nobody needs some fancy ass model to figure out how answering on those questions might indicate suicidal tendencies. Ive had two friends commit suicide out of nowhere in the past year, family didn't expect it, and nobody who was close to them expected it. Its them that are hard to spot and stop not the obvious "I'm on drugs, my father's abusive, I'm socially withdrawn" people.

“ Girls are more likely to have thoughts but less likely to act, pretty well known, so maybe it doesn't make sense to treat thoughts as seriously for them but this model makes it the same.”

I don’t see any sources to back up your claim, so here is some food for thought that says otherwise: https://www.cdc.gov/mmwr/volumes/70/wr/mm7024e1.htm

“ During February 21–March 20, 2021, suspected suicide attempt ED visits were 50.6% higher among girls aged 12–17 years than during the same period in 2019; among boys aged 12–17 years, suspected suicide attempt ED visits increased 3.7%.”

So girls are actually acting on it at an alarmingly higher rate than boys (up ~50% from last year due to the pandemic).

The fact that they end up in ED is actually proof that girls act on their suicidal thoughts in non-effective ways. Boys who shoot themselves in the head, or jump from the window, or in front of the train, they don't end up in ED, they go straight to the morque.

A girl would typically swallow some pills, and then call their best friend to say godbye, giving time and opportunity to help them.

Girls don't want to die ugly. They don't people to see them with a shot up face or a mangled body even after they are dead. They hope that if they take pills or slit their wrists they'll just look peacefully asleep when they are discovered.

I have never heard of any case where they've called their friends.

Not called yes but everybody ive know has sent goodbye text message or snapchat.

My understanding is that girls are more likely to attempt suicide, while boys are more likely to die from suicide.

As a sibling comment mentions, a big factor in this is differences in how the groups go about in their respective attempts.

I am also not aware, however, of statistics on long term effects of survived attempts of suicide.

> Like legit half of it is drug use questions.

That's on purpose:

"Communities That Care (CTC)[1] is a program of the Center for Substance Abuse Prevention"

"The Center for Substance Abuse Prevention (CSAP) aims to reduce the use of illegal substances and the abuse of legal ones.[14]"

That doesn't make it any better at predicting anything.

Ya i wasnt shit talking the study, i was saying its a bad dataset to use in this case.

Just ask them. This algo uses survey results for its input anyway, so why not just ask people directly? I find the notion of "predicting suicidal thoughts" to be overreaching and struggle to stay off of the slippery slope of thinking how dangerous such a thing could be institutionally.

There's a normed and valid survey that asks the question already. OQ Measures' YOQ 2.0 Youth Questionnaire is widely used in behavioral health programs. Answers to these questions are combined to form a sub scale score that a professional may use in implementing feedback informed treatment.

Q21. I have hurt myself on purpose (for example, cut, scratched, or attempted suicide). Q41. I think about suicide or feel I would be better off dead.

AFAIK, most people who are at risk of suicide do not have someone in their lives who "can just ask them directly", that's a key factor in why they're suicidal in the first place. So you'd specifically want a means of detection for people who no-one is in a position to directly ask.

But this method already relies on survey answers, so it's not helpful in this regard.

Ignoring the study design, etc, 91% doesn't seem great for something like this. How many of those other 9% are false positives? Being mislabeled as suicidal by an algorithms that administrators or other people in power place their trust could create more problems. I certainly don't want an algorithm to tell others, including myself, what I feel (or what it's guessing I feel).

More than just that, what about the parents?

My dad’s reaction to me having suicidal thoughts was say I was being ungrateful. It wasn’t until my thirties I was diagnosed bipolar.

Some of that story is on my blog. https://kayode.co/blog/4106/living-with-psychosis/

I'll come in with a hot take on this. Whether a computer can predict these types of things should be interesting to computer scientists/software engineers only.. Humans are very good at knowing who feels these things and or at least will be better than computers for the foreseeable future.

For the most part the problem isn't knowing there is an issue with a child, the problem is getting the social, cultural, political, and individual will and skills to address it in a meaningful and productive way. Identifying the issue doesn't do a lick of good if we have no way to address it. This is, for the most part, the real problem. Have you talked to your kids about suicidal thoughts/mental health in a salient way that gives them space to approach it? More importantly, have you talked to them about how to talk to their friends who are having these problems?

This is really the broader issue, mental health issues like suicide, addiction, and depression are solveable if and only if we are able to deal with the ugly discussions on a cultural and personal level. Singling broken people out by algorithm won't help us there.

> The sample population consisted of more than 179,000 high school students living in Utah and participating in the Communities That Care (CTC) Youth Survey from 2011-2017.

Utah is not known for its religious diversity. I wonder how this would look over a different population.

There could also be a difference between what teens say and what they feel. I though there had been some studies that suggested that suicide wasn't as tied to external factors as people thought. Depression is a mental illness, it can happen to people with good families and good friends.

Well, and not just low religious diversity in Utah. Low diversity on most measures.

If you could dig up sources on studies I'd greatly appreciate it. I never pursued psychology to the end after getting a bachelors in it, but mental health is still I cause I care about and would like to see improvements.

Sucks to be in that 9% that ends up strapped to a bed in a hospital somewhere for no reason.

Great to be one of the doctors that gets the bed filled and paid for "for a reason". An invalid reason, but very valid money. People forget who these studies are for, to "help make policy", but really to help practitioners. The purpose of these papers is: firstly to justify the existence of these professions and institutions, to get more money to their initiatives. The purpose is not to protect or help children.

You can tell by the content of studies in this field. They are always about detection. "Customer" acquisition. Throwing more kids into the system. However the child protection systems ... don't protect children very well. Not so long ago there was an article on this site "We are sending more foster children to prison than to college".

https://web.archive.org/web/20210321010610/https://www.kansa...

There are many articles that observe that psychiatric treatment of suicidal kids, and adults for that matter, actually increases the odds they die through suicide instead of reducing it. Which makes putting more people into these situations, will cause more death (and a lot more misery).

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5710249/

Clearly anyone who had the welfare of kids in mind would only ever want any action taken that would improve the situation of the child, and would absolutely refuse to act if it didn't. But we know very well if these treatments help or not. They don't.

Therefore any policy that actually seeks to help children would massively reduce children going into the system to the cases that are absolutely unavoidable, and would prevent any expansion of the system until they are very sure kids do well in this system.

These people don't just don't do that, they actively advocate for more resources TAKEN AWAY from caring for these children, and going into finding more of these children to be treated. That tells you that not only do they not care at all about children's welfare, but they're willing to make children's situations worse to advance other goals.

And that should be a very strong reason to tell them to go away, and never come back.

Combine this with mental health laws like Florida, where a mere suggestion that you might be at risk for self harm is enough to take away your rights, and we can just start proactively institutionalizing people for being statistically likely to self harm. Yay progress?

It seems there's a general trend towards this sort of thing. Sort of like how red flag laws are popping up everywhere - use civil proceedings that bypass protections under the criminal system, don't tell the person about the hearing so they can't defend themselve in court, and don't even require any real evidence just vague unverified statements by others.

Authoritarianism has many forms, of which this is merely another.

> Combine this with mental health laws like Florida, where a mere suggestion that you might be at risk for self harm is enough to take away your rights

Citation?

https://en.wikipedia.org/wiki/Baker_Act

The letter of the law says a judge or doctor has to initiate it, but the doctors often do it after a suggestion from a parent. I knew people in high school that were stuck in there for days without consent.

Thank you, I couldn't remember the name.

> adolescents without a father in the home were 72.6% more likely to have suicidal ideation than those that did

I wonder why this is, what the opposite (lacking mother) percentage is, and why that wasn't so salient.

The census data has the numbers, 91-92% of kids live with both parents or a mother. 4% live with only their father, 4% live with neither. With those numbers I think it'd be hard to get a significant sample.

https://www.census.gov/library/stories/2021/04/number-of-chi...

This was a big survey, I think it would have been significant. They had 41,434 responses without a "father living in the home", so by the census numbers you'd expect ~8200 responses without a mother in the home. They also didn't consider kids without either parent, which should have been equally significant. It does seem like a weird omission - I found what I think is the survey they used (https://web.archive.org/web/20200707085026/https://www.commu...) and the study authors did have this information, they just didn't use it for some reason.

The wording they use, "father in the home", seems like it covers kids without any parents or only grandparents as well as kids with single mothers, so the phrasing seems pretty misleading.

Could the feeling that you lack control of your circumstances be a contributing factor, and could the clumsy application of this kind of algorithm further diminish a teen's sense of control?

The documentation for the Shap tool they used has a nice big warning against doing that (clumsily applying the algorithm predictions): https://shap.readthedocs.io/en/latest/example_notebooks/over...

I wonder how many read that section?

The first thought that crossed my mind was that I wouldn't like to be a false positive. It could be pretty nightmarish. Once you are flagged as a suicide risk, denying that will only raise further suspicion and pressure to "admit the truth".

Here is a concise summary of the classification methods and results of the paper:

> Modeling was conducted with a variety of machine learning algorithms using the scikit-learn library: K-Nearest Neighbors, Naive Bayes, Logistic Regression, and DecisonTreeClassifier, as well as XGBoost and LightGBM. Models were trained and tuned on the train and validation splits, evaluating only once on the held-out test set for our final numbers.

Direct link to ROC figure:

https://journals.plos.org/plosone/article/figure?id=10.1371/...

Ignoring for a moment the statistical hand-waving and terrifying 1-in-10 probability of being labeled suicidal—and the corresponding restrictions of freedoms—I was a bit taken aback by the high predictive power of yelling and arguments at home.

My parents never—literally never—raised their voices or argued with one another. I knew other families were different from television and books, and when I married into a family that yells liberally I took it in stride as just another mode of communication. Now that I have a kid, I'm wondering—for those of you with a spouse with a temper—what are some strategies for de-escalating arguments and yelling?

Zillow can't even predict house prices with an algorithm, and now we are going to tackle suicide?

How does being Gay in the LDS church contribute I wonder?

Reply on news.ycombinator.com