Hacker News Reader

Comment by ivansavz | Hacker News Reader

This is a great paper. Very long, but worth every bit of it. BTW, here is a recent blog post about the paper: http://duboue.net/blog27.html

One of the key insights I took away was the importance of using out-of-sample predictive accuracy as a metric for regression tasks in statistics—just like in ML. The standard best practices in STATS 101 is to compute R^2 coefficient (based on data of the sample), which is akin to reporting error estimates on your training data (in-sample predictive accuracy).

IMHO, statistics is one of the most fascinating and useful fields of study with countless applications. If only we could easily tell apart what is "legacy code" vs. what is fundamental... See this recent article https://www.gwern.net/Everything the points out the limitations of Null Statistical Hypothesis Testing (NHST), another one of the pillars of STATS 101.

View on HN

> The standard best practices in STATS 101 is to compute R^2 coefficient (based on data of the sample), which is akin to reporting error estimates on your training data (in-sample predictive accuracy).

That's not the best practice at all. It ain't even standard because adj R^2 exist to penalized coefficient cheating and if you cheat on your degree of freedom. And on top of that we got other penalization functions other than R^2 such as AIC and AAIC.

All of them are just for comparison between similar type of models and it's not suppose to be use for generalization test for unseen data. We're taught CV too and it is of statistic invention. And also taught about training set and test set with CV.

If you want to debug for test data you have to do CV anyway for imbalance data. So this is comparing apple to banana with R^2 vs out of sample.

You can see all of this in the book that applied statistic uses for linear regression, Applied Linear Statistical Models by Kutner & et al.

> The standard best practices in STATS 101 is to compute R^2 coefficient (based on data of the sample)

> Null Statistical Hypothesis Testing (NHST), another one of the pillars of STATS 101

Best practices of statistics and what is taught in Stat 101 are not remotely the same thing. The problems with R^2 and NHST have been well documented they've been argued against for decades by the statistics community. But actual statisticians make up a small proportion of statistics practitioners, as statistics is the backbone of nearly all modern science. What gets taught in Stats 101 is not the "foundations of statistical best practices" so much as "a system of guidelines and rules of thumb that have been simplified greatly for the sake of the lowest common denominator". To make things worse, non-statisticians seem to over-estimate their statistical knowledge and prowess after having taken a handful of introductory stats courses more than any applied field I know of.

Only now after the magnitude and pervasiveness of the replication crisis has begun to be recognized by the broad scientific community are people starting to realize what many statisticians have been pointing out for years.

> The standard best practices in STATS 101 is to compute R^2 coefficient

Any regression class which doesn't teach about overfitting is incomplete. R^2 is still a very useful measure, however, if you've come up with a parsimonious model and want to describe its descriptive performance.

Reply on news.ycombinator.com