Hacker News Reader

Comment by banku_brougham | Hacker News Reader

The concept of no members of a group fitting into the average range for all observations is reminiscent of the 'curse of dimensionality'. Can anyone with a data science background make the connection here?

Secondly, this seems to explain why everyone hates autocorrect.

View on HN

Yes, this can be made precise: For example, consider a uniform distribution on a d-dimensional cube with side length 1. Only a (1/2)^d fraction of the mass is within 0.25 distance of the average in every coordinate.

In high dimensions, almost all the mass is "near the boundary" in at least one coordinate.

Or as one of my professor would say (after showing the calculation for n-dimensional spheres): "and that's why infinite dimensional oranges are all skin!"

This must be a common saying. I've heard it from every lecturer I've ever had for machine learning and sampling techniques.

I suspect it comes from statistical physics, as it's a very common approximation when you're in the thermodynamical limit, which basically means that the phase space has a very large number of dimensions.

It's undoubtedly pithy. (sorry!)

Lecturers get together in a staff room early in the morning and rehearse the delivery corny jokes in unison, the way teams of shop clerks in Japan go over phrases like "irasshaimase" and "omatase shimashita".

Weirdly enough, even the proportion of points within a distance of 1/2 to the midpoint becomes really small for higher dimensions.

takeaway: With enough dimensions everyone is weird

Or -- if you think everybody is the same -- then you aren't looking hard enough!

Bad news for advertisers who want to use a zillion targeting factors -- and yet somehow not be disappointed by the minuscule audience that results ...

Yeah, in this case if the 10 traits are independent and the chance of falling in the "average range" is 1/3 for any one trait then the probability of any one soldier falling the the average range for 10 traits is (1/3)^10 = 1/59049.

That's assuming that they're independent, though, which is certainly false. I'm curious by how much.

Why is it certainly false?

The simplest example I can come up with is the weight - the density of humans is probably constant for practical purposes and therefore the weight must be positively correlated with your body dimensions. It is also pretty obvious that humans, or tissue for the matter, does not grow in a strongly directional manner. If you gain weight more or less all dimensions not constraint by your skeleton will increase, i.e. you will probably not end up with a wide but flat or narrow but thick belly.

Sure, but weight gain in adulthood doesn't cause you to get taller.

That is way I said not constraint by your skeleton - or what ever else. The point is that at least some dimensions are obviously correlated, not that all dimensions have to be correlated.

>weight gain in adulthood doesn't cause you to get taller //

Not even a tiny bit, like because of fat feet or more fat on the head? Not trying to be facetious, just seems huge people have fat feet and that there's likely to be a little fat on the bottom of the foot. Mind you that's going to be countered by compression of the spine, so there could be a inverse correlation?

I wouldn't say density is a constant. Muscle weighs more than fat for the same volume, as does bone. Same height, higher muscle or denser bone structure, and you can have wildly different weights. Makes simplistic measurements like BMI pretty laughable - look at pretty much any athlete, who would run high over-weight or obese according to BMI.

That doesn't matter, humans are a lot of water and the density of humans seems - after a quick search - to be essentially within the density of water plus or minus ten percent. In this context that's constant enough for me and definitely way to small to possibly offset the variability in body weight.

But you might naturally assume a strong correlation between the middle range of each dimension - the interesting conclusion is that it's not nearly as strong as our intuition suggests.

But even if they are correlated that does not necessarily imply that being average on one dimension makes it likely to be average on other dimensions. I can well imagine that correlations conspire against you, i.e. that being average in one dimension makes it likely to be non-average in another dimension.

That appears to be the case, but is counter-intuitive enough that it took the air force years to realize it.

I assume there are developmental genetics that apply to a whole organism. Everything on an elephant has to be big and everything on a mouse has to be small. But then there probably is little correlation between localized development genetics. The result is that within the 5% to 95% range the correlation between the size and shape of developmental features is pretty much zippo and simple non-correlated probability dominates.

That is true, but this can be just as much an issue in 1 dimension. Consider that the average person has roughly 1 testicle. I think it's about understanding distributions, and joint distributions are a big part of that.

It doesn't have to be a high-dimensional space. 0% of a donut is near the average.

> The concept of no members of a group fitting into the average range for all observations is reminiscent of the 'curse of dimensionality'. Can anyone with a data science background make the connection here?

? It would seem like you have made the connection already. Although I don't see it.

Reply on news.ycombinator.com