Hacker News Reader

Comment by sevensor | Hacker News Reader

This comes at a good time! I was just talking with my spouse about why you put n-1 in the denominator when you calculate variance. I always liked "because you used up a degree of freedom calculating the mean," but I feel like that's kind of hand-wavy. (Obviously, I'm not a statistician.)

View on HN

You can argue either way. dividing by n is the maximum likelihood estimator, but dividing by n-1 is the unbiased estimator. Depends on what you like. Most people prefer unbiased estimators when they are available.

A mathematician once gave me a convincing argument for dividing by n+1 instead, but I have since forgotten his reason.

The argument is that this factor can minimize the mean squared error of the variance estimate, at the cost of some bias. In general, the small correction to the lead factor depends on the fourth moment. For a Gaussian, you get 1/(n+1).

This is an example of a “shrinkage estimator”, which comes up a lot - introduce some bias but get a smaller MSE. For more, see: https://en.wikipedia.org/wiki/Bessel%27s_correction

What does it mean for it to be "unbiased"? What does it mean to "use up" a degree of freedom?

I don't mind if things just can't really be explained intuitively because they are fundamentally technical, but your explanation and the parent's both do this thing where it sounds like it's explaining things in plain common language, but isn't actually because it isn't clear what those plain words mean in this context.

"Unbiased" means the error is on average zero.

Unbiased means that if I draw infinitely many random samples from a population and average a statistic (in this case standard deviation) across all the samples, the answer will be the statistic computed from the population itself. If one divides by n instead of n-1, the estimate for standard deviation will be be (n-1)/n too small. One reading this might think, "Wait! We're going to infinity so the ratio converges to 1." That's true if the size of each sample also goes to infinity but not if we draw millions of ten item samples.

As for using up a degree of freedom, the easiest way to build intuition for why this is a useful concept is to think about very small samples. Let's say I draw a sample of 1 item. By definition the item is equal to the mean so I receive no information about the standard deviation. Conversely, if someone had told me the mean in advance, I could learn a bit about the standard deviation with a single sample. This carries on beyond one in diminishing amounts. Imagine I draw two items. There's some probability that they're both on the same side of the mean, in that case, I'll estimate my sample mean as being between those number and underestimate the standard deviation. Note that I'd still underestimate it even with the bias correction, it's just that that factor compensates just enough that it balances out over all cases.

A simple, concrete way to convince yourself that this is real is to consider the standard deviation of a variable that has an equal probability of being 1 or 0. The standard deviation is 0.5. But if we randomly sample two items, 50% of the time they'll be the same and we'll estimate the standard deviation as zero. The other 50% of the time, we'll get the right answer. Hence, our average is half the right answer (n/(n-1)=2/1). The correction makes the standard deviation double what it should be half the same while remaining zero in the other cases. This also suggests why dividing by n is referred to as a the maximum likelihood estimator.

Thanks! This is helpful!

This is very helpful, especially the example at the end, thanks. I think the difficult part to understand is that dividing by n leads to an estimate that is somehow too small. The intuition tells you that dividing by n would just give you the true average.

Asymptotically, all the same. If sample size is small, it's a bad estimator anyway.

You guys might like to watch this video:

https://www.youtube.com/watch?v=dq_D30kyR1A

https://www.youtube.com/watch?v=ANsVodOu1Tg

For those wondering, this is what the resource at hand says about the matter:

"To be precise, when this mean is calculated, the sum of the squared deviations is divided by one less than the sample size rather than the sample size itself. There's no reason why it must be done this way, but this is the modern convention. It's not important that this seem the most natural measure of spread. It's the way it's done. You can just accept it (which I recommend) or you'll have to study the mathematics behind it. But, that's another course."

Could you link to which of the pages in OP's book (webpage) this is?

I tried in vain to go to the likely one and still couldn't find where the variance gets introduced!

Reply on news.ycombinator.com