Hacker News Reader

Comment by mchahn | Hacker News Reader

The "paradox" is that if player's batting averages, when high, get worse predictions and low averages are predicted to get higher, then these predictions work better than just guessing they will stay the same.

This doesn't seem like a paradox to me. Rather it seems kind of obvious. If the statistics are anything like a random walk then random walk theory (usually revisiting the starting point) would predict this.

View on HN

I think the article by Richard Samworth lays out the paradox better. The whole article is worth a read, but here's the paradox part " To give an unusual example to emphasise the point, suppose that we were interested in estimating the proportion of the US electorate who will vote for Barack Obama, the proportion of babies born in China that are girls and the proportion of Britons with light-coloured eyes. Then our James–Stein estimate of the proportion of democratic voters depends on our hospital and eye colour data! " <http://www.statslab.cam.ac.uk/~rjs57/SteinParadox.pdf> Surely that's paradoxical!

The OP's post is an outstanding exposition of James-Stein estimators though, so thanks for the post. There seems to be lots of connection between these and doing linear regressions with regularisation in machine learning.

Yep, there's a link with regularization and also with informative priors – James-Stein works so well because across an incredibly wide range of scenarios, a parameter estimate of infinity is not nearly as likely as a parameter estimate of 0, yet that's what ordinary least squares linear regression assumes.

Fixed link: http://www.statslab.cam.ac.uk/~rjs57/SteinParadox.pdf

Yeah, I think you're right. Everyone expects a reversion to the mean -- even the batters themselves. It becomes a self-fulfilling prophecy.

But maybe it's more than that. The will power to overcome our intrinsic limits eventually runs out. We can only focus on the ball so long before our thoughts wander. And at some point below the average of our capabilities, we get tired of taking a break and start to perform again as well as we know we can.

Reply on news.ycombinator.com