Hacker News Reader

Comment by Bootvis | Hacker News Reader

Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.

View on HN

I think the problem is that certain subfields use different terminology to mean similar or identical concepts. For example, while I'm in software, I tend to hear the terms sensitivity and specificity. They are historically medical terms. They aren't identical to recall/precision, but I think you can derive one set from the other.

The fundamental thing to know is the confusion matrix. There are about a dozen terms for various descriptors of the matrix, but they all can be calculated if you know the confusion matrix. The Wikipedia page has a great table to describe them all:

https://en.wikipedia.org/wiki/Confusion_matrix

You can see from that that sensitivity and recall are the same thing, but specificity and precision are not.

That certainly seems likely and a good thing to keep in mind when you're giving or taking an interview!

It's literally the first thing you learn in data science / machine learning coursework about evaluating model performance. It would probably be better to ask the candidate to whiteboard a set of metrics for evaluating model performance rather than ask for the definition of a pair of words, but the concept is practically the for-loop of data science.

Edit: note that I'm not saying you need this to add roi as an analyst for a business!

For those downvoting this comment -- it's absolutely true that model performance is discussed at length early in ML courses (usually in the context of the bias-variance tradeoff).

My only quibble would be that precision + recall are one set of evaluation metrics applicable to classification tasks. Modelers can absolutely use other loss functions.

Additionally, precision/recall do not map nicely to regression problems, so people use other metrics (RMSE, MAE, etc.).

I haven't taken a lot of data science classes but I'm not sure that's true. If you start with linear regression the mean squared error would make more sense. I actually searched through "The Elements of Statistical Learning" and the word 'recall' is not used in this sense at all.

The jargon does vary by subfield and community, along with the actual measures used (sometimes it's just a different name, but sometimes practices are different as well). Precision/recall are terms from information retrieval that migrated into the CS-flavored portion of machine learning, but are not as common in the stats-flavored portion of ML, in part because some statisticians consider them poor measures of classifier performance [1]. Hence they don't show up in the Hastie/Tibshirani/Friedman book you mention, which is written by three authors solidly on the stats side of ML. It does occasionally mention some equivalent terms, e.g. Ctrl+F'ing through a PDF, I see that in Chapter 9 it borrows the sensitivity/specificity metrics used in medical statistics, where sensitivity is a synonym for recall (but specificity is not the same thing as precision). It looks like the book more often uses ROC curves, though, which have their own adherents and detractors.

[1] This paper is the one that most often gets cited as background by people who don't like recall/precision as metrics: http://dspace2.flinders.edu.au/xmlui/bitstream/handle/2328/2...

People don't pay for linear regressions. They pay for discrete things: what is my best option among my three clear courses of action. Linear regression can be a tiny piece of a larger argument in favor or against one option or the other, but that alone doesn't make money.

That's obvious but not at all what I responded to in my post.

I responded to the claim that ML courses start with the definition of precision and recall. In my admittedly limited experience those courses start with linear regression and mean squared errors. After that, there is so much generalization possible and that doesn't include precision/recall.

You make money by solving someone's problems, making money by stating definitions is only done on TV quizzes.

That's OK. The article was talking about somebody interviewing for a search-related position (where precision and recall are usually what you are optimizing for). I guess they might be called differently in econometrics?

Reply on news.ycombinator.com