When the U.S. air force discovered the flaw of averages | Hacker News Reader

When the U.S. air force discovered the flaw of averages

thestar.com

471 pointshecubus103 comments

The concept of no members of a group fitting into the average range for all observations is reminiscent of the 'curse of dimensionality'. Can anyone with a data science background make the connection here?

Secondly, this seems to explain why everyone hates autocorrect.

Yeah, in this case if the 10 traits are independent and the chance of falling in the "average range" is 1/3 for any one trait then the probability of any one soldier falling the the average range for 10 traits is (1/3)^10 = 1/59049.

That's assuming that they're independent, though, which is certainly false. I'm curious by how much.

Why is it certainly false?

The simplest example I can come up with is the weight - the density of humans is probably constant for practical purposes and therefore the weight must be positively correlated with your body dimensions. It is also pretty obvious that humans, or tissue for the matter, does not grow in a strongly directional manner. If you gain weight more or less all dimensions not constraint by your skeleton will increase, i.e. you will probably not end up with a wide but flat or narrow but thick belly.

Sure, but weight gain in adulthood doesn't cause you to get taller.

That is way I said not constraint by your skeleton - or what ever else. The point is that at least some dimensions are obviously correlated, not that all dimensions have to be correlated.

>weight gain in adulthood doesn't cause you to get taller //

Not even a tiny bit, like because of fat feet or more fat on the head? Not trying to be facetious, just seems huge people have fat feet and that there's likely to be a little fat on the bottom of the foot. Mind you that's going to be countered by compression of the spine, so there could be a inverse correlation?

I wouldn't say density is a constant. Muscle weighs more than fat for the same volume, as does bone. Same height, higher muscle or denser bone structure, and you can have wildly different weights. Makes simplistic measurements like BMI pretty laughable - look at pretty much any athlete, who would run high over-weight or obese according to BMI.

That doesn't matter, humans are a lot of water and the density of humans seems - after a quick search - to be essentially within the density of water plus or minus ten percent. In this context that's constant enough for me and definitely way to small to possibly offset the variability in body weight.

But you might naturally assume a strong correlation between the middle range of each dimension - the interesting conclusion is that it's not nearly as strong as our intuition suggests.

But even if they are correlated that does not necessarily imply that being average on one dimension makes it likely to be average on other dimensions. I can well imagine that correlations conspire against you, i.e. that being average in one dimension makes it likely to be non-average in another dimension.

That appears to be the case, but is counter-intuitive enough that it took the air force years to realize it.

I assume there are developmental genetics that apply to a whole organism. Everything on an elephant has to be big and everything on a mouse has to be small. But then there probably is little correlation between localized development genetics. The result is that within the 5% to 95% range the correlation between the size and shape of developmental features is pretty much zippo and simple non-correlated probability dominates.

That is true, but this can be just as much an issue in 1 dimension. Consider that the average person has roughly 1 testicle. I think it's about understanding distributions, and joint distributions are a big part of that.

> The concept of no members of a group fitting into the average range for all observations is reminiscent of the 'curse of dimensionality'. Can anyone with a data science background make the connection here?

? It would seem like you have made the connection already. Although I don't see it.

Yes, this can be made precise: For example, consider a uniform distribution on a d-dimensional cube with side length 1. Only a (1/2)^d fraction of the mass is within 0.25 distance of the average in every coordinate.

In high dimensions, almost all the mass is "near the boundary" in at least one coordinate.

Or as one of my professor would say (after showing the calculation for n-dimensional spheres): "and that's why infinite dimensional oranges are all skin!"

This must be a common saying. I've heard it from every lecturer I've ever had for machine learning and sampling techniques.

I suspect it comes from statistical physics, as it's a very common approximation when you're in the thermodynamical limit, which basically means that the phase space has a very large number of dimensions.

It's undoubtedly pithy. (sorry!)

Lecturers get together in a staff room early in the morning and rehearse the delivery corny jokes in unison, the way teams of shop clerks in Japan go over phrases like "irasshaimase" and "omatase shimashita".

Weirdly enough, even the proportion of points within a distance of 1/2 to the midpoint becomes really small for higher dimensions.

takeaway: With enough dimensions everyone is weird

Or -- if you think everybody is the same -- then you aren't looking hard enough!

Bad news for advertisers who want to use a zillion targeting factors -- and yet somehow not be disappointed by the minuscule audience that results ...

It doesn't have to be a high-dimensional space. 0% of a donut is near the average.

The thing that strikes me here is how much we repeat the mistake of Norma (taking the mythical average body and assuming that differences mean problems) today in talking about people's minds.

I started using computers before it was fashionable, before my fellow nerds started being worth billions and ending up on magazine covers. It's hard to describe now how much our difference was seen as wrong, as the sign of a problem.

I have to wonder how much other natural differences gets medicalized. I know I have friends who take drugs for "insomnia" even though the only problem they experience is that they don't always get the "average" night of eight hours uninterrupted sleep. (Which anyhow is a modern invention. [1]) When I was in mourning after my mom died, a few buttinskys suggested I talk to my doctor about antidepressants, even though actual experts thought I was doing fine. And I worry about the number of schoolkids who have their differences medicalized because, in effect, they are inconvenient for overburdened teachers using industrial-age models of education.

I have no solution here, but I definitely find it troubling.

[1] For more, see this podcast, especially the "Til Morning is Nigh" segment: http://backstoryradio.org/shows/on-the-clock-4/

Modern psychology does not care about average it cares about capable of functioning in society. If you can't avoid say screaming in public that's a problem not just a non issue. Arguably the tolerances might be low, but it's hard to argue with the basic premise.

Granted doctors assume if your talking to them you have a problem. But that's arguably a reasonable prior.

PS: The extremes of behavior get very far out there, notmenjoying talking to people is fine, not being able to so communicate is an issue.

>If you can't avoid say screaming in public that's a problem not just a non issue. Arguably the tolerances might be low, but it's hard to argue with the basic premise.

People are prescribed stuff for much much less than "screaming in public", including feeling stressed in BS high stress jobs, being "hyper-active" at school, etc -- with "modern psychology" more often than not siding with the normality of abnormal and abusive societal norms (as it always did. Science is conservative. Not so long ago you got medical treatment for being gay, for example).

>Granted doctors assume if your talking to them you have a problem. But that's arguably a reasonable prior.

Doctors also prescribe and over-prescribe all kinds of BS to people who don't need it, and get all kinds of pharma percs for doing so.

Heck, they'll go even to doing un-needed operations, compared to that, prescribing BS is nothing:

https://www.google.gr/search?q=uneeded+operations+scandal

I don't know that I agree with your characterization of modern psychology, and I am anyway talking more about psychiatry and its chemical interventions.

But even if I granted your point, you're still demonstrating one aspect of the problem I'm talking about. You suggest that any time there's a mismatch between a person and the society they find themselves in, there's a problem with the person. Rather than with society, or with their expectations or what "functioning" means, or with some other aspect of the situation.

The same thing happened with the pilots. The assumption was that the planes were fine, so any issue must be with the pilots. But the planes were constructed around a mythical "normal" person, and I am suggesting we construct society and our expectations of its participants around a "normal" person.

>How much we repeat the mistake of Norma (taking the mythical average body and assuming that differences mean problems) today in talking about people's minds.

Makes me wonder which is the perfect text editor that everyone should use, is it emacs or vim?

Every post on HN is, at most, six degrees of separation from a emacs vs. vim debate. In this case, it was a mere two.

While hindsight is 20:20, parts of this should have been more obvious.

> Daniels generously defined as someone whose measurements were within the middle 30 per cent of the range of values

> Daniels discovered that if you picked out just three of the ten dimensions of size ... less than 3.5 per cent of pilots would be average sized on all three dimensions.

30% raised to the third power is 2.7%. Basic probability. I guess everyone assumed there would be heavy clustering instead of largely independent variables?

Assuming a normal distribution instead of a uniform one isn't that outlandishly unreasonable.

He didn't assume a normal distribution. It wasn't "within ±15% of the mean". It was "the middle 30% of the range". This means excluding 70% of the population for each factor, regardless of the distribution. Finding a "normal person" was impossible from the beginning with that approach.

The original assumption before Daniels was that the data would follow a normal distribution and so there was an "average" pilot.

He showed that was wrong.

I can't tell what you mean. In some sense (not a very good one), the data must follow a normal distribution, as the sum of normal distributions is itself a normal distribution. But that would imply that you could sensibly add values like "circumference of upper arm" and "length, ankle to knee", which you really shouldn't be doing.

- If you measure values from a 10-dimensional space, what would it mean for them to be "normally distributed"? Do we norm them by distance to the center of the space, or what? Does that approach preserve the nice properties of normals?

- If this concept exists, did Daniels or anyone related to the project know that? Is it really what they were looking for?

- If this concept exists, were the pilots in fact normally distributed?

Saying "the data should follow a normal distribution" without having a definition of "normal distribution" that allows for multidimensional points immediately implies that all body measurements are correlated with all other body measurements at plus or minus 1, which I'm certain nobody believed. And the individual measurements are normally distributed.

I this case they may not have been. Given the US Airforce was selecting pilots with measurements within certain bounds their measurements were probably a truncated normal.

You are looking for https://en.wikipedia.org/wiki/Multivariate_normal_distributi...

"the middle 30% of the range" is ambiguous. I interpreted it as mean ± .15 * [max - min]

Also it's only impossible from the beginning if you assume the values are uncorrelated. That's an even worse thing to assume than the existence of an 'average person'.

Only if the variables are independent, right? Because I would expect a tall guy to have longer arms than I have.

> The Aero Medical Laboratory hired Daniels because he had majored in physical anthropology, a field that specialized in the anatomy of humans, as an undergraduate at Harvard. During the first half of the 20th century, this field focused heavily on trying to classify the personalities of groups of people according to their average body shapes — a practice known as “typing.” For example, many physical anthropologists believed a short and heavy body was indicative of a merry and fun-loving personality, while receding hairlines and fleshy lips reflected a “criminal type.”

Also, for what's it worth, this was during (or at least not too far removed from) a time period where researchers were trying to ascribe all sorts of random characteristics to people based on whether they were "Negroids", "Caucasoids", or "Mongoloids".

This says a lot to me about how people thought about physical characteristics back then. If researchers assumed male-pattern baldness was associated with a "criminal type", then it's not a huge stretch to imagine that researchers assumed there was a "pilot type" with a large number of clustered or heavily correlated characteristics.

Lombroso et al.

Worth noting that this phenomenon is typically described as “medicalization.” Othered groups medicalized include women, African Americans, homosexuals, and many other groups. The medicalization of Jewish ancestry cropped up and was amplified during a not-too-distant past.

That is over-simplification. First, the samples are not randomly selected. The selection process tended to lean toward means, based on the article. Second, the variables are not independent. They are strongly correlated. The selection process indicated that the probability of having a pilot with long legs but short arms was close to zero.

> They are strongly correlated

Uh, no. What the study shows is that people assumed these measurements would be strongly correlated, but they were not.

When only 3.5% of people fall into the middle 30% on 3 variables, that is strong evidence that they are mostly independent.

joint distribution of 2 normal distributions isn't necessarily a normal distribution in 2 variables. Independence would be sufficient condition for that, yet obviously there is no independence between various measurements of a human body.

https://en.wikipedia.org/wiki/Multivariate_normal_distributi...

I was thinking it was a Venn diagram, where none of the bubbles had complete intersection with the others.

"This bubble represents all pilots between 5'6" and 5'11" in height. And here's the bubble that represents those with 40" chests. Note how they barely intersect."

Which is funny, because the problem seems to arise exactly because the variables are independent. If they were dependent then one could imagine all sizes correlate with height, for example .

Wow! This has implications for human genetics (a study that originated partly in considering human body size measurements like those described in the article) that I think most popular writers on human genetics have not taken into account. There was an article posted to Hacker News earlier this week that suggested, surely falsely in my opinion, that human genomes can be optimized for intelligence to such a degree that we won't even be able to estimate IQ scores for future individuals with the same kind of tests that we use now. I rather doubt it. Because of pleiotropic effects of genes, almost surely there isn't a gene shuffle that will massively increase human intelligence, but rather just strategies (as likely to be environmental as genetic) to improve average intelligence within the range already found in the worldwide human population.

Similarly, imputation of "race" by genome testing depends crucially on assuming that the average genome is informative, but for any trait of interest, a person who by both genome testing and known historical ancestry is categorized in some "race" category might have any degree of variance from "average" in the trait found in the whole human genome. It will be exciting to follow up on larger and larger data sets on these issues as human genomics projects continue.

Similarly, imputation of "race" by genome testing depends crucially on assuming that the average genome is informative,

This is a nonsensical mischaracterization of both modern genetics and the theory of high dimensional vector spaces.

The phenomenon the article is discussing is the fact that the mass near the center of a normal distribution approaches zero as the number of dimensions goes up, and that most of the mass lives a distance sqrt(N) from the origin. This does not imply that a) you can't have gaussians or other distributions which are separated from each other or b) that projections of one of these distributions can't have a different distribution than a projection from the other.

Please go read Foundations of Data Science, chapters 1 and 2 before expounding on this topic further: http://research.microsoft.com/en-US/people/kannan/book-no-so...

Can you clearly state what, precisely, you think this article proves about either classification, intelligence augmentation or group differences in specific traits?

I can talk about some of this, if only because of some of the ethical dilemmas facing me and genome testing as of recent.

Racial Groups as described by a geneticist vs say, a government doing polling on its citizens are different animals.

Goverments == mostly sociological. An example would be Hispanic in the US, where growing up data collection would ask if someone was Hispanic, and now it asks if you are white vs non-white Hispanic. Meanwhile in say, Meanwhile, if someone crossed the border to Mexico, there is no such thing as Hispanic - you can be White, Mestizo (Indigenous-European hybrid), Indigenous, and Other. It is totally possible to live on the border of the US-Mexico, have reasons to commute across the US-Mexican border, have citizenship to both countries, and have totally different answers on your census depending on what country asked you about your race/ethnicity, because as can be clearly seen, the way that question is asked is different in mexico and the US to begin with.

Let's talk about being Mezito as an idea. From a geneticist's point of view, it doesn't exist. (or at least, not yet, and it is unliekly to any time soon) Which is how 23 and me and buzzfeed manages to get this Gif off of one of Buzzfeed's employees, who very clearly feels he is half mexican (aka mezito) https://img.buzzfeed.com/buzzfeed-static/static/2016-01/21/1... In other words, there are no clusters that define "mexican"/mezito

In order to have such clusters, you need to have long histories in one area, histories of inbreeding, and other major causes to make mutations pop selectively. Even with those mutations popping selectively, you also need those mutations/genes to be very trait specific in most cases/ultra selective. Otherwise, you are looking at junk.

So, one of the reasons ashkenazim are testable as ashkenazim is because they have a very long history of inbreeding. So do the japanese compared to other asian groups, same with the Finnish, especially if you are Saami.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2575505/

Still once these ibreeding/long history facts start to subside, you end up with whatever is the strongest trait set in a pure mendelian fashion. It is why there are plenty of ashkenazim who definitely decendants of the founder populations of ashkenazim who don't carry other sets of Ashkenazi marker genes, whereas there are ashkenazim where you can't tell who do carry these other genes. The other random sets come from other places (sex..mutations..) and are bred out with enough time. (Otherwise there would be many many more questions about how I exist, genetically speaking, since I should not be able to metabolize alcohol in about 1.5 hours...)

However, most traits are not one gene == one trait. It's how the music plays together, especially in concert with epigentic factors. For many things involving "intelligence" (or actually lots of things for humans), we're looking at a non-mendellian inheritance pattern, including Genomic imprinting issues ( https://en.wikipedia.org/wiki/Genomic_imprinting ). So while some gene related to "intelligence"/"thinness"/"insert something here" might be more observable among certain groups/populations, it doesn't mean it doesn't exist among other populations and some other factor, genetic, epigentic, or otherwise, is suppressing your view.

This is why if/when you get involved with genetic research on humans, the ideal is to get the entire family (as much as possible) involved in tests - even if the original candidate comes from a human genetic isolate polpulation. There are otherwise broader concerns about power and sample size in the study, because you might be looking at something which is actually something else.

http://www.nature.com/gim/journal/v4/n2/full/gim200210a.html

(but, hey, what do I know...I had to ask geneticists and people doing research in this area when I found out my genome is worth something accidentally...)

Great post, thanks! You use 'Mestizo' once, and then 'mezito' several times. Is this slang, a repeated typo, or a term of the art? Google doesn't find much for it.

Mestizo means mixed Amerind European ancestry, exactly like mulatto means mixed African European ancestry. They're both from colonial Spanish racial categories.

https://en.wikipedia.org/wiki/Mestizo

My phrasing was unclear. I'm familiar with "Mestizo" --- it's "mezito" that I'm wondering about. Urban Dictionary suggests that "mezito" is just a typo, but the author uses it consistently and perhaps separately than Mestizo.

Deep in the search results are occasional other people who appear to be using it intentionally: https://www.stormfront.org/forum/t53845/

I'm wondering if like "cholo" it has particular connotation: https://en.wikipedia.org/wiki/Cholo

Incidentally, I was recently fascinated to learn that some consider "cholo" to come from Nahuatl: http://www.etymonline.com/index.php?term=Cholo

repeated typo/spelling things like they are sounded to my ear rather than how wikipedia tells me to spell things :)

This is old news for geneticists. Since Mendel and Punnet we've known that a.) traits don't average or "mix" from parent to child and b.) individual traits don't assort randomly.

In a more modern context, the Human HapMap project (http://hapmap.ncbi.nlm.nih.gov) is working on mapping out just how many different assorting groups exist among human genes.

You might like this article:

http://blog.watsi.org/netflix-and-global-health/

Great article.

Still the Netflix recommendations are awful and contradict my taste whatever I tried to help it understand the movies/series I liked and disliked. As result I subscribe one month, this month, to see House of Cards and then unsubscribe till next season. Maybe this average algorithm works for illnesses and mortality, IMHO it is not biased enough to my non-US taste/culture to be of any use for Netflix recommendations.

Actually if anything this phenomenon would increase the probability of the super IQ thing. Intelligence isn't determined by one gene, but many thousands. Just like there are very few people who happen to be exactly average on dozens of different traits, there are very few people who happen to have the absolute best possible combination of genes to get max intelligence.

That is, assuming there is some optimal combination of genes, it gets exponentially more unlikely any individual has it, the more genes there are that affect intelligence. Just like you would be unlikely to flip 1,000 heads in a row. In fact flipping even 600 heads out of a thousand, is exceedingly unlikely.

My company Treemetrics has been challenging the forest industry for method used for measuring forests. They use the average tree size as the method for valuing a standing forest. From our data we see forests with thousands of trees where only a handful of average trees actually exist. Very helpful article for me to explain the flaw of averages.

This is incredibly poignant in a world where our designers and engineers are moving to provide users less options and less configuration, deeming their ideal designs, meant to meet 'most users needs', is superior.

exactly!

This was a very interesting article. And I can’t speak to the statistical methodology described. But I think it paints too broad a brush in terms of why the Air Force fatality rate (or “Class A Mishap Rate” as it’s called) decreased over the decades. Quick side-note – I’m a former USAF pilot and studied USAF history a bit.

The short version is – technical details like adjustable cockpits certainly played a part. But the bigger driver was the ever-increasing culture of safety, mishap prevention and leadership accountability.

Safety: things that were common place “back in the day” have been banned for many decades in today’s Air Force. These include traveling without appropriate weather checks, acrobatics maneuvers at low altitudes and unauthorized airspaces, and flying without enough sleep.

Mishap Prevention: incredible resources are poured into scientific investigations on what exactly happened in each incident so that A) we can learn from it and B) all pilots can be briefed on the mistakes made so that they can save their own lives.

Leadership Accountability: if you are a senior leader at an Air Force base with a fatality, you have a large issue on your hands that needs to be handled with extreme attention to detail. Lapses in safety are not looked favorably upon at any level. I wasn’t in the USAF back in the day. But it’s fairly obvious that there was less punishment meted out when mishaps occurred compared to when they happen now. Different times – yes. But overall fatalities were just accepted as “something that happens.”

The safety culture is all-pervasive and common in today’s military. But there were different attitudes back in the ‘40s. So while many of today’s pilots gripe about the excesses of safety culture - and I was one of them – most pilots know deep down that it’s a good thing. I wouldn’t want to return to the old school days where crashes were just the cost of doing business.

Thought this would be about http://www.johndcook.com/blog/2008/01/21/selection-bias-and-...

Articles like this remind me how rational and practical the military is. I know so many brilliant military minds, yet war itself seems outdated and plagued by so many emotionally irrational decisions. I can't reconcile those two thoughts.

How can people inside the military be so incredibly smart, yet still think it makes sense to ... you know, kill people's friends and families and not expect them to become terrorists.

What am I missing? It can't just be greed. Military industrial complex. The military minds aren't smarter than that? I struggle with it.

Momentum.

Also, it's easy to paint yourself as "the good guy with all the government sponsored guns and military contracts" who will "keep the other good people safe" because "you're better, smarter, more capable and more honorable than them." It's a very sexy and lucrative mindset, so it shouldn't be a surprise that it's pervasive.

Do military people disagree with your analysis about killing ?

I was really dismayed when Colin Powell argued for invading Iraq. Yes, there are military people who get it. But just look what's happening.

I guess they have to think, what is the alternative? We can't change our role in the Middle East. We can't just stop supporting Israel for example.

I suppose yes, they tried Camp David. I guess I just want everyone to cease fire.

Maybe the overpowered people resorting to terrorism, I do hate that word, have to think there is a positive outcome from the cease fire. If they cease firing and life doesn't improve, why did they cease firing? At least firing alleviates the frustration of it all. They think they are hurting those who are hurting themselves.

But life for the Palestinians never improves no matter. It just keeps getting worse and worse and worse for them. The people with the power have to stop exerting it over others and selfishly taking more.

I had a buddy in Afghanistan and he said their way of life is 500 years old. I mean, I think okay, if they don't want to progress, that doesn't mean we should use the tools of our progress to take from them so we can progress more.

I think at some point we have to stop betraying people.

High star generalship is more politics than military ability at this point.

http://www.theatlantic.com/magazine/archive/2012/11/general-... for one discussion of this.

a) war a very unforgiving environment - bad policies become apparent very quickly under pressure

b) the things you are against ("kill people's friends and families and not expect them to become terrorists") are often policies not under military control. If the government doesn't want to spend the money or annoy its allies enough to solve the root causes, there's not much a soldier on the ground can do except mitigate the damage (and yes, that often includes actions that perpetuate a bad status quo)

Blaming the military for stupid shit that the military does is like blaming police for stupid shit that police do. It might make us feel better, but it ignores the root of the problem, and therefore perpetuates the stupid shit. That is, we, the society that puts the military into impossible situations ("topple governments in the Middle East, and then hang around a couple of years to make sure that ends in order, peace, and prosperity for all") and the police into impossible situations ("you seem to have violence under control, here are three or four more basic human drives we'd like you to thwart in capricious fashion") are the root of the problem. So long as we allow ourselves to be distracted from that, we can expect more stupid shit.

...and yes, of course the military-industrial complex is complicit in our ongoing distraction, but they only hire the pundits and media personalities. We're the ones who listen to them.

As a vet myself, you seem to have a view of the military that's straight out of a Tom Clancy novel (that everyone is extremely overly competent and highly educated and intelligent). "Articles like this remind me how rational and practical the military is. I know so many brilliant military minds"-no, the military is full of people, a lot of whom are as at least as fucked up on many levels as people outside the military.

The military is just the apotheosis of bureaucracy. Bureaucracies live off of data, often to a fault, when whatever they're after can't be reliably measured.

I can't speak for the entire military[1], but I've known a few "brilliant military minds" and they rarely, if ever, talk in terms of killing. The military's goals are usually things like area denial, disrupting lines of communication, limiting access to materiel and supplies, degradation of morale, etc.

These aren't just euphemisms. If you're trying to win a war solely by killing the enemy, you're going to be in for a looooong and bloody war. In fact, there's long been a thinking in the military that "a dead soldier removes one soldier from the field, but a wounded soldier removes two".

What makes the military, and war, seem "outdated" or downright "despicable" comes down to, I think, two things:

1. War is often what happens when two sides have let long lingering issues fester to the point that dialogue is not possible. In other words, the only thing anyone wants less than to throw the first punch is to be unable to throw the second.

2. Yes, military industrial complex. Specifically, the MIC has muddled the "goals" and abstracted them away behind many layers of "weapons systems" and "advanced tactics". That is, if you're a general attempting to disrupt lines of communication, and you have to decide on committing the lives of your soldiers and potentially taking the lives of your enemy, you might consider a battle plan that minimizes loss of life. If you're that same general and Raytheon (or Lockheed or Honeywell or...) offers to sell you the CommsRuptor 7000 that will take out enemy communications at the press of a button (and a signature on a check for $300M), you might not adequately question the impact on lives. War used to be about loss of blood and treasure. Lately it seems to be more about treasure and blood...

[1] Especially not the "grunts" that see front-line action. I've heard, anecdotally, that enlisted soldiers are often trained in such a way that they enter war zones with raw blood-lust. Whatever your opinion on this practice, I took from your question that you were more interested in the thinking of the decision-makers, i.e. commissioned officers.

I wouldn't say "raw bloodlust" is accurate. I'd say the life of a serving infantryman is so boring, repetitive, and mundane that getting the chance to use your training in combat will look very pleasing to almost anyone. Especially on deployment.

The idea that killing a person's friends of family turns them into a terrorist, and that this is the ultimate source of terrorism and terrorists, is an oft-repeated but highly unsupported claim. It's ultimately a cudgel wielded by advocates of particular political ideologies, not serious students of terrorism and counterterrorism.

"the average man" is a very 19th Century Determinist concept. There may also be other 19th century Determinist concepts lurking around in military doctrine.

Those are frequently the troublesome ones that don't fit actual measurement.

At least in the US, there was no standing army until the 20th Century.

I'm pretty sure averages used in the wrong way affect many more fields than any of us can realize today.

That's basically the entire premise behind the Black Swan meme.

For anybody getting a message that says "Please try loading the page again after activating mobile data or connecting to a Wi-Fi network." = it's not your connection, despite what the page says. The mobile site, http://m.thestar.com/ , is giving that message on desktop too and it comes from the site itself.

Has nobody heard of quetelet?

No, but thanks for bringing him to my attention! It's amazing how someone like that so long ago could have such an impact on so many fields, while today, it takes teams of people to win the Nobel prize in a single field.

It's a testament to how much we know today relative to yesterday.

I think we had a discussion here on HN not long ago but I can't find it.

edit:

Found the article:

http://www.theatlantic.com/business/archive/2016/02/the-inve...

comments: https://news.ycombinator.com/item?id=11155889

Is it just me or does this sound oddly familiar to machine learning recommendation systems?

ie: show an item based on the average of all the other items that everyone else has looked at.

Yeah, I was thinking of the same thing. It's not my field professionally though, so I am not sure what the relevant conclusion would be for machine learning though (or if the comparison holds at all).

I don't think that's quite correct.

Recommendation engines normally work by taking the things you favor, then looking at who else favors those things, and deducting that you're all likely to have shared interests.

Yeah I'd assume the simplest recommendation engines work on correlation. A recommendation engine working on average would be complete garbage, you'd be better off picking recommendation at random.

Quote emcq's response above:

> If you're thinking about the cold start problem when you dont have any information about a user, yes it's possible that your overall statistics is a combination of many subpopulations that doesn't really fit anyone very accurately but there are ways around this as well.

I'd say a better approach (but may have user experience penalty) would don't even decide for the user, ASK. For example, take Flipboard / Quora as an example, you are asked to choose some topics to follow at the beginning. Assume research/data show 80% of the users are software engineers and 90% of them always pick "technology" as a topic they want to follow, would you rather show technology as one of the top five in the list of topics to choose? There's actually a lot of experiments you can do from a simple selection/survey process. I personally can't stand at going through pages to find something relevant, but I am also surprised to find things I never thought would be interesting to follow if I weren't present the options at all / or earlier.

Perhaps for a naive recommendation system, but quality ones don't do this.

For example if you employ 1-nearest neighbors, you will actually return a sample that is guaranteed to reproduce another user's preferences. You could of course sample over more neighbors, smoothing results, at the cost of more lookups and at some point in the limit you do see reduced performance due to too many points becoming similar and having this average of everyone being bad for everyone sort of thing. Most ML approaches account for this and will only look at user/item clusters where the assumption is that a user can be sampled from a user cluster and an item can be sampled from an item cluster. Locally you may have error incorrectly capturing the clusters but in practice this works fairly well and you dont have a one size fit all model.

If you're thinking about the cold start problem when you dont have any information about a user, yes it's possible that your overall statistics is a combination of many subpopulations that doesn't really fit anyone very accurately but there are ways around this as well.

So in short - a naive ML recommender that perhaps you get out of a textbook would certainly face these issues but the are addressable in production with an "industry" grade recommender system and will lead to better performance. I don't have firm numbers but in A/B/C tests compared to other black boxes I've seen 20-30% better performance when you start to address some of these issues.

This is a great example.

It doesn't take a lot of dimensions, though. I've seen many system workloads where the average latency was not the latency of any of the requests.

Thinking about this and the wisdom of crowds makes my brain feel fuzzy.

If the environment was homogenized enough this would no longer be an issue.

analogous anecdote: if you've ever tried to drive a sports car in a seating/steering/pedal/shifting position that isn't tailored to your body, it feels really awkward and it's very difficult if not impossible to drive fast with precision.

i would guess that in a sports or race car, everything needs to be within a few millimeters of where it "should" be in order for it to feel right. multiply the speeds and divide reaction times by 4-5x and i can see how you could easily crash a plane with a tiny margin for error.

Reminded me of

“Be together. Not the same”

~Android :)

I don't see the problem. The fact that no person fit the model perfectly doesn't mean that the model was a poor choice.

The question shouldn't be whether any customers fit the cockpit perfectly, but how many customers do and do not fit absolutely. You build the legroom so that most legs will fit, and headroom so that most heads will fit. Then as many people as possible shall fit. Everyone will have some dimension that isn't perfectly accommodated, but few should be rejected. The fact that nobody fits perfectly doesn't take away from a design that reasonably accommodates as many people as possible.

A cockpit is not a suit. It's a communal chair/workspace meant to be used by various persons over many years. The metric of a good design should be how many/few people are so out of standards that they cannot work properly in the space. This is a perfect metaphor for hiring. Look only for that 'prefect fit' and you won't hire anyone, or you end up with hiring the only applicant to get past the roboreader. Seek a broader standard and you'll find plenty of good people even though mr perfect never appears.

>I don't see the problem. The fact that no person fit the model perfectly doesn't mean that the model was a poor choice.

Did you skip the part about people dying?

> A cockpit is not a suit.

The opposite is true. To fly an airplane with precision, you need to "strap the airplane on."

That means sitting in the position with the best visibility and best reach inside the cockpit, even a Cessna 172.

(In a 172, you should sit high enough to be able to see each of the rivets on top of the cowling.)

Yes, but the seat does adjust. It isn't the case that the cockpit must be tailored to every person. Each Cessna, 747 or f-16 seat is practically interchangeable amongst the type. These aren't WWI biplanes with nothing more than a cushion. Even then, the cushion height, the prime dictator of eye level, is adjustable.

I don't know about postwar planes, but at least some of the WWII war birds had notoriously poor seat adjustments. For example BF-109 had a seat that doubled as armored fuel tank. It's bit tricky to add adjustments to such heavy piece of furniture, so they didn't.

The thing here, just like you pointed out, is not to tailor the seat for average nor individual, but a range that suits most individuals.

> Yes, but the seat does adjust.

It does now. It didn't then.

Did you read the article? That's the entire point of it!

The seat adjusts because of the study described by the article...

"Out of 4,063 pilots, not a single airman fit within the average range on all 10 dimensions."

"If you’ve designed a cockpit to fit the average pilot, you’ve actually designed it to fit no one."

Reply on news.ycombinator.com