Hacker News Reader

Comment by lalaithion | Hacker News Reader

Note that this is not some sort of unexpected effect due to humans not behaving mathematically ideally; this is what happens when your intuitions about 1, 2 and 3 dimensions are applied to higher dimensional spaces. Consider the goal of being in the middle 50% of a random value on one dimension; you have a 50% chance. But if you have two dimensions, and you want to be in the middle 50% for both of them, it's a 25% chance. And if you have 3, it's a ~12% chance. This already is counterintuitive, but when you ramp it up to 10 dimensions, it's a ~0.01% chance. That means that if you have a thousand people, only one of them (on average) will be in the middle 50% of all of them. Even the praised end state of the air force, with 9 dimensions and a required support of 90% of individuals of each dimension, implies that only 38% of individuals will be supported overall. (Granted, human dimensions aren't wholly uncorrelated.)

View on HN

Great explanation. This is an example of the curse of dimensionality. An intuitive way to think about this is that each dimension you add is another “chance” for one of them to fall outside of the range.

Another way to think about it is the often-cited unit n-dimensional sphere. If you were to uniformly sample points from within this n-dimensional sphere, as n increased, the proportion of points lying near the surface of the sphere would increase.

Numberphile has a good one on how these things can get crazy quickly:

https://www.youtube.com/watch?v=mceaM2_zQd8

That last bit seems to be the Crux of why this is so surprising -- being in the middle 50% on some dimension correlates positively with being in the middle 50% on the other dimensions, rather than each dimension being independent as in your calculation. It's difficult for me to reconcile that, for the 400 in the study within 5% of average height, each was a standard deviation away from the mean in some other measurement.

It’s two different effects. First being within a range is not the same as being at the center of the range. Someone 4% above average height should have ~50% of their dimensions larger than that. Similarly someone at 4% below average should have around 50% of their dimensions below that. At best someone in the exact middle of the range only has so much buffer to work with.

Second the correlation is less significant than your assuming. My legs are the same length as some people a full foot shorter than I am.

Every time I get my height measured by a new doctor, they feel the need to tell me that my legs are short for my height.

I don't recall ever having my height measured. Is that a U.S. doctor thing?

Visits often start with a basic vitals check, but as an adult they've been content to simply ask my height, while measuring my weight.

Sure, but being in the middle 10 percentile in height(or some other dimension) would serve to "normalize" the sample; so 400 people are close to the center of the larger range. Despite being close to the middle of the range, some dimension is far from the center of it's range.

Yes, the correlation is not as strong as I would assuming -- that was really the point of my comment. You are a sample size of one, so your anecdote doesn't mean much. However, based on this work, apparently almost everyone has a similar anecdote: after normalizing for height, there is another common dimension which is "unusually" large or small.

> would serve to "normalize" the sample

The slope of a bell curve near it’s center is almost flat. This means you end up with a fairly uniform distribution when looking at values near the median. Which makes outliers within that range more common than intuition suggests.

I find it more intuitive to to discard the spatial intuition and think in terms of events. You want something in the X% norm? Think about it like a X%-heads biased coin flip. If you flip it enough times, it'd be shocking to get all heads. Even if the coin flips (dimensions) are correlated, many coin flips (dimensions) means at least some tails (dimension outside the norm).

Should note that the study seems to be working on an extremely narrow range, the 30% in middle to quote the article. So measurements in the 35-65% percentile from what I understand.

In layman's term. It's so narrow that there are more people 1 inch off than there are people within the expected height. It's crazy.

Probability is talking in terms of standard deviations nowadays. They are selecting less than half a standard deviation, it's hyper selective. I'm curious how many people would fit the norm if the study was looking at 1 standard deviation. Surely a lot more.

For reference. Selecting the 30% on six metrics is keeping less than 0.01% of participants. Selecting the 68% (one deviation) on six metrics is keeping 10% of participants. It's night and day. Should be even more in practice because measurements are correlated.

This comment was what made this article make sense to me.

Just to add to add to the mathematical intuition here (please correct me if I'm wrong): if you're thinking of it as a unit line/square/cube then total n-dim area is 1^n, and the portion in the 50% range is (1/2)^n, where n = number of dimensions. So that should simplify to 2^(-n).

Note this works out to 10th dimension as 0.09765% or 0.1 person per 1000.

ETA: As one of the comments below points out, you can also model it as a binomial distribution.

Probability of getting all heads, given p=0.5 is (n!)/((n-k)!k!) / 2^n. Since n=k since we're looking to get all heads at all times, this also simplifies to 2^(-n).

P(simultaneously within middle 50 % of n independent distributions) = 0.5^n = 2^(-n), like my parent comment says.

But interestingly, every relevant comment here either got something wrong about the final frequency or percentage, or corrected the wrong thing. (Parent comment, grandparent, 2 aunts and 1 cousin.)

0.5^10 = 2^-10 = 1/(2^10) = 1/1024 [i.e. exactly 1 person per 1024] ~= 0.00097656 ~= 0.09766 %.

Hah, you are correct. I even had access to numpy and still managed to get this wrong.

Or to work the example in the article: Being in the middle 30% of any three traits has a 2.7% chance (matching the quoted 3%). For ten traits it's a 1e-6 chance.

This is only true if each dimension is independent and identically distributed random variables, which is doubtfully true for humans. That said, the overall intuition is probably more correct than incorrect...

They are neither fully independent nor fully dependent. It falls somewhere in the middle. The same will apply, just in a slightly less dramatic effect.

But surely the engineers were who were inventing jets and rocket guidance systems and radar understood probabilities of independent events. The point, which the author almost misses completely, is that they dramatically overestimated how correlated the measurements were.

Beyond that, while there's no accounting for taste, I find it to be an appallingly bad article. It's poor man's Malcolm Gladwell.

Understanding the probabilities of independent events is a different thing than identifying them.

It is very common for engineering projects to encounter problems caused by an issue which was understood but was not identified. Most engineering failures fall into this category. Humans simply make mistakes.

1 of 10,000 is .01% right?

Was going to make the same comment, should be 0.1 people out of 1000.

nit: 0.01% is 1 in ten thousand.

how did you derive 38%?

0.9^9

thanks

Reply on news.ycombinator.com