Hacker News Reader

Comment by scottfr | Hacker News Reader

I'm a huge fan of R^2 and you should be too.

The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model.

All the figures and findings in the post are completely reasonable given that definition (the linear model performing better in the first case is due to the high variance of the author's data, other samplings would lead to the reverse result).

View on HN

I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) depending on the model. Just using R^2 blindly for most tasks you would work on can lead to choosing an incorrect model.

I think the main take-away from the post is to better understand the correct measure for model fit for your specific data. For example, if you are forecasting a time series with stationary demand, mean average deviation might be the best measure of model fit, but it the case where there is seasonality with a trend, the RMSE would be a better measure [1].

[1] http://robjhyndman.com/papers/foresight.pdf

> I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model)

I don't mean to be rude, but that is definitely not what he is saying. There are two important things I'd like to clarify:

- It is wrong to call the alternative measure "the RMSE". The alternative that the article was proposing was a made-up measure called E^2 which measures the distance from the true model.

- The author is not suggesting that people use E^2 instead of R^2 for any case. In fact, in almost all cases it is impossible to use E^2, because calculating it requires you to know the true model, and if you know the true model it would be very unlikely that you'd be wasting your time measuring other models.

The author makes it clear that E^2 isn't really to be considered an alternative when he called it the "generally unmeasurable E^2".

I think we are saying the same thing in different words, and I might be confusing "an alternative" with "a comparison". John compares R^2 with E^2, but RMSE can be considered an alternative to using R^2 in certain cases.

If you go back to the first line: > People sometimes use R^2 as their preferred measure of model fit.

I think the post is going over why R^2 is not recommended as 2 is not only a measure of the error, but it includes a comparison with a constant model. John defines E^2 as a comparison metric which measures how much worse the errors are than if you used the true model.

Going back to a metric for determining model fit, RMSE/MSE/MAD are all alternative measures of model fit and are useful depending on the dataset.

Reply on news.ycombinator.com