Hacker News Reader

Comment by mjburgess | Hacker News Reader

One important caveat to all these methods is that the central limit theorem must hold for the sample means and this is an empirical condition, not something you can know statistically.

Another important caveat: many things we want to measure are not well-distributed to allow the CLT to hold. If it doesnt, the bulk of statistical methods don't work and the results are bunk.

Many quantities follow power-law distributions which would require trillions+ data points for the CLT to do its magic, ie., for the sample means of set A to be statistically-significantly different from set B would require 10^BIG if the property measured in A/B is powerlaw distributed.

Now, even worse: many areas of "science" study phenomena which is almost certainly power-law distributed, and use these methods to do so.

View on HN

In fact, the CLT is remarkably robust to distributional assumptions. Examples where it breaks down (e.g., non-finite variance) are comparatively rare, even if there are "many" of them.

As with all things statistical, judgment is required.

> Examples where it breaks down (e.g., non-finite variance) are comparatively rare, even if there are "many" of them.

I would beg to differ. They are absolutely not rare. [0] One of the most famous people in probability theory even claimed that it's the ones with finite moments that are rare in practice. (But I couldn't find this quote back, I thought it was Poisson or Laplace)

It's even worse. Many distributions where the CLT does apply, require so many samples for them to actually work, that it does not really apply in practice anymore. Any skew in your data blows up the amount of samples you need to find things like the empirical mean.

[0] Chapter 3.4 https://arxiv.org/ftp/arxiv/papers/2001/2001.10488.pdf

> One of the most famous people in probability theory even claimed that it's the ones with finite moments that are rare in practice. (But I couldn't find this quote back, I thought it was Poisson or Laplace)

That sounds more like Kolmogorov than like Laplace. Someone more interested in mathematical issues than in practical questions.

I don't think this is entirely true.

Most heavy-tailed distributions make CLT convergence slow, and heavy-tailedness is relatively common.

Many methods in finance and pharmacology, just to give two examples, have had spectacular failures because they ignored that point.

> many things we want to measure are not well-distributed to allow the CLT to hold

I guess that may be true for some values of "many" and "we" but most things we want to measure have finite variance.

for some examples of "not-most" things - https://en.wikipedia.org/wiki/Cauchy_distribution#Occurrence...

I'm not sure I'm fully understanding your point. Is it that constructing confidence intervals using t-statistics is inappropriate for a lot of real data that isn't distributed somewhat normally?

It's their point, and it's a good one, but I think they're somewhat overstating how common power-law data is; it probably varies a lot by field of study. And at least the logarithm of a power-law variable can help bring it back closer to the world of sanity. Plus, there are plenty of fields where nonparametric tests of medians are accepted standard practice.

You can turn most issues into powerlaws by recursing a reasonable risk distribution over it.

So suppose we ask, what is our confidence in X? (rather than X); and then, what is our confidence in the model by which we give confidences in X (ie., the model risk); and so on...

In practice, what we want to model is the appropriate confidence, not an actual prediction (bunk). So we are very often screwed.

Statistics is an illusion.

Reply on news.ycombinator.com