Hacker News Reader

Comment by PaulHoule | Hacker News Reader

I like those charts showing they get an AUC of 0.8 or so, my own content-based recommender gets 0.72 or so on a typical day with a very simple model, I think it could get 0.75 with mainly data-oriented tweaks. I was thinking a better model could get me into the high 70's but the fuzzy nature of the recommender problem means I am never going to get into the 90's.

I would like to see more about how they are formulating the problem, I know there is work lately in "sequential recommendation" that is focused on generating a sequence rather than scoring content items, I'd like to learn more about that.

View on HN

> the fuzzy nature of the recommender problem means I am never going to get into the 90's

Right. If your AUC is in the 90's you usually either have a bug, or a predictive task that isn't very challenging.

Some tasks are hard, others are easy. I get in the low 60’s trying to predict if a headline will get a lot of votes on HN. Of course this is going to be bad because the same article can get submitted several times, hit it big once and not hit it big the other times. The content of the article matters too but the model can only make a wild-assed guess. I get in the high 70’s trying to predict if a post with a lot of hits gets a lot of comments, it is a cleaner problem. I get about 90% trying to predict if a post winds up ‘dead’, I found up HN is cleaning up a lot of spam posts when they go in but I know it is terrible overfit because it says an arXiv paper which has no chance of being flagged has a 30% chance of flagging.

For some other like “is this an astrophysics or materials science paper?”, getting 0.95 or better was straightforward in 2005.

AUC being “area under curve” or something else?

Correct, it is a performance measure for ML algos (https://www.sciencedirect.com/science/article/abs/pii/S00313...)

Yep. See https://en.wikipedia.org/wiki/Receiver_operating_characteris...

If you had to pick one number for judging classifiers that is the one.

"Accuracy" is a really bad one because it is not meaningful w/o careful thought about the problem. For instance if you have a diagnostic system for a disease that has a .01 prevalence, you get 99% accuracy if you say nobody has a disease.

Practically a person might want to operate a classifier with different thresholds depending on how bothered they are by false positives vs false negatives, the ROC shows what choices are have available with a particular and the AUC of the ROC gives you one number that characterizes the quality of the classifier overall. A system with an AUC under say 0.55 hasn't really learned anything at all, at 0.6 it is showing signs of life, looking at those curves I'd say Tik Tok is beating me solidly. In this case it is limited by the problem being fuzzy (something I thumbs up today might get a thumbs down tomorrow), something up in the 0.95 level could be attained for something like "is this article about the outcome of a sports game?"

Reply on news.ycombinator.com