Hacker News Reader

Comment by nightcracker | Hacker News Reader

Very sloppy statistics. A lot of explanatory analysis on the model is done but the model is hardly justified in being predictive in the first place.

The paper (https://journals.plos.org/plosone/article?id=10.1371/journal...) [does not mention the sensitivity or specificity of the model at all, only mentions a '91% accuracy' number]* on a biased dataset (where the number of suicidal cases is oversampled and non-suicidal cases are undersampled), without even mentioning exactly how much they over/undersampled.

* I missed the ROC curve on page 7. However it's not clear if this ROC curve was computed on the under/oversampled dataset or the original.

View on HN

> The paper [url snipped] does not mention the sensitivity or specificity of the model at all

There's a full ROC curve in Figure 2. Just eyeballing it, it looks like they get both sensitivity and specificity in excess of .9 in the top left corner (I didn't try to measure it precisely).

It would certainly be helpful to have more information about the over/undersampling.

I did miss that, oops. That said, it's again unclear if that curve is pre- or post- biased sampling. If it is on the original data it is quite good, but with a heavy biased sample fairly meaningless.

If it's not made clear, post-sampling analysis is the safe assumption.

"You have to oversample your minority class so your data is balanced" is an urban legend that needs to die. It is never necessary, unless you are using extremely outdated model fitting methods, and even then, it would only be needed in training. It is completely unnecessary in evaluation, and metrics on biased data are going to be biased (obviously!).

If I hear this in an interview, I'm going to assume you do data science by blindly copying random blog posts.

Agreed. ROC AUC is fairly robust to over or undersamping a class in determining if your classifier is predictive.

See my sibling comment. Suppose you have data with 0.95 positive class and 0.05 negative class. You can achieve high ROC AUC with the classifier that blindly predicts the positive class. It may be "predictive" (after all 0.95 of the data is positive), but I would hesitate to praise such a classifier.

While I agree with your overall sentiment, it is important to understand the relationship between class (im)balance and ROC curve performance. A very short article which does a great job explaining this is [0]. There are, of course, other performance metrics that are appropriate in the presence of class imbalance, such as precision-recall curves, so I wouldn't go so far as to say "metrics on biased data are going to be biased". Some metrics can correct for the class imbalance bias. Others can't.

[0] https://www.researchgate.net/profile/Jake-Lever/publication/...

What do you mean by class imbalance bias?

Even if this works, the only question it answers is which questions on the survey correlate with considering suicide. The title is deceptive in that regard.

Reply on news.ycombinator.com