Hacker News Reader

Comment by halflings | Hacker News Reader

Ignoring what a p-value is does not mean that you don't know statistics. p-tests are not some inherent statistical property, they're just a useful model for significance. People coming from a CS background most likely didn't have to deal with p-values, but they can still be good at linear algebra or bayesian statistics.

(not sure I can defend somebody that does not know what precision/recall are)

View on HN

Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.

I think the problem is that certain subfields use different terminology to mean similar or identical concepts. For example, while I'm in software, I tend to hear the terms sensitivity and specificity. They are historically medical terms. They aren't identical to recall/precision, but I think you can derive one set from the other.

That certainly seems likely and a good thing to keep in mind when you're giving or taking an interview!

The fundamental thing to know is the confusion matrix. There are about a dozen terms for various descriptors of the matrix, but they all can be calculated if you know the confusion matrix. The Wikipedia page has a great table to describe them all:

https://en.wikipedia.org/wiki/Confusion_matrix

You can see from that that sensitivity and recall are the same thing, but specificity and precision are not.

It's literally the first thing you learn in data science / machine learning coursework about evaluating model performance. It would probably be better to ask the candidate to whiteboard a set of metrics for evaluating model performance rather than ask for the definition of a pair of words, but the concept is practically the for-loop of data science.

Edit: note that I'm not saying you need this to add roi as an analyst for a business!

I haven't taken a lot of data science classes but I'm not sure that's true. If you start with linear regression the mean squared error would make more sense. I actually searched through "The Elements of Statistical Learning" and the word 'recall' is not used in this sense at all.

The jargon does vary by subfield and community, along with the actual measures used (sometimes it's just a different name, but sometimes practices are different as well). Precision/recall are terms from information retrieval that migrated into the CS-flavored portion of machine learning, but are not as common in the stats-flavored portion of ML, in part because some statisticians consider them poor measures of classifier performance [1]. Hence they don't show up in the Hastie/Tibshirani/Friedman book you mention, which is written by three authors solidly on the stats side of ML. It does occasionally mention some equivalent terms, e.g. Ctrl+F'ing through a PDF, I see that in Chapter 9 it borrows the sensitivity/specificity metrics used in medical statistics, where sensitivity is a synonym for recall (but specificity is not the same thing as precision). It looks like the book more often uses ROC curves, though, which have their own adherents and detractors.

[1] This paper is the one that most often gets cited as background by people who don't like recall/precision as metrics: http://dspace2.flinders.edu.au/xmlui/bitstream/handle/2328/2...

People don't pay for linear regressions. They pay for discrete things: what is my best option among my three clear courses of action. Linear regression can be a tiny piece of a larger argument in favor or against one option or the other, but that alone doesn't make money.

That's obvious but not at all what I responded to in my post.

I responded to the claim that ML courses start with the definition of precision and recall. In my admittedly limited experience those courses start with linear regression and mean squared errors. After that, there is so much generalization possible and that doesn't include precision/recall.

You make money by solving someone's problems, making money by stating definitions is only done on TV quizzes.

For those downvoting this comment -- it's absolutely true that model performance is discussed at length early in ML courses (usually in the context of the bias-variance tradeoff).

My only quibble would be that precision + recall are one set of evaluation metrics applicable to classification tasks. Modelers can absolutely use other loss functions.

Additionally, precision/recall do not map nicely to regression problems, so people use other metrics (RMSE, MAE, etc.).

That's OK. The article was talking about somebody interviewing for a search-related position (where precision and recall are usually what you are optimizing for). I guess they might be called differently in econometrics?

I usually tailor my questions to whatever the person said they did. If they say they did search, I ask about precision recall. If they discuss hypothesis testing, I'll ask about p-values or other approaches.

I'd happily take a Bayesian answer if they preferred that, but that hasn't happened very often.

I can't imagine how you could learn Bayesian stats w/o starting with a basic inference class, which will cover p-values at length.

I did Bayesian stats without ever discussing p-values. Not al classes discuss them. (my background is ML, where p-values are not as useful as say in biology or any field relying on experiments with control groups)

You're certainly not going to study Bayesian statistics without knowing (or at least having studied) what a p-value is.

Actually p-values are used way less often in Bayesian statistics than frequentist ones. The latter rely on statistical tests more.

Bayesian stats tend to use likelihood ratios or Bayes factors instead of p-values for hypothesis testing.

The trick in all cases is that you're comparing to expected results given some prior distribution. Most people use a dumb prior (e.g. Gaussian) and then they're confused when the numbers make no sense as data is multimodal or heavy tailed, thus mismodelled.

I studied statistics - my point was that statistics is taught in a linear manner, starting with distributions and hypothesis testing (p-values) and then move onto more advanced treatments like Bayesian stats.

That happens in statistics programs. However, I have a ML-heavy minor in CS, and based on the ML course contents at our CS dept I've seen, I'm not sure if the all their CS majors go through the the full canonical statistics curriculum, nor that they were intended to. At least the ML courses had quite much introductory probability and statistics as far as ML applications were concerned, so I understood the implication was they didn't assume that the students would have already done the similar stuff in statistics (though it certainly helped), and I can't remember a single mention of p-value there.

And then there's this, that even if your intro to probability course everywhere covers the classic statistics with p-values and hypothesis testing and frequentist confidence intervals and so on, you are not necessarily going to use them that much. I calculated some p-values and other tests with R for some example datasets a couple of years ago and never seen them since in coursework, everything we've done after that has been more or less fully Bayesian. The concepts are still fresh[1] in my mind mostly because I read some statistics blogs, such as Andrew Gelman's [2]. The irony is that Gelman does not exactly love frequentist framework, he just mentions its concepts often enough.

[1] or not totally forgotten

[2] http://andrewgelman.com/

Reply on news.ycombinator.com