The diversity trumps ability theorem | Hacker News Reader

The diversity trumps ability theorem

pnas.org

The punchline, from the abstract: "We find that when selecting a problem-solving team from a diverse population of intelligent agents, a team of randomly selected agents outperforms a team comprised of the best-performing agents. This result relies on the intuition that, as the initial pool of problem solvers becomes large, the best-performing agents necessarily become similar in the space of problem solvers. Their relatively greater ability is more than offset by their lack of problem-solving diversity."

It's not always true, of course, that a randomly selected group will outperform a group of top performers. But the result does suggest that it's worth thinking hard about the tradeoff between diversity and ability, and that sometimes the former is more important than the latter.

Scott Page again? See here for a deconstruction of one of his previous, simpler arguments.

http://www.overcomingbias.com/2007/04/the_error_of_cr.html

This one looks more complicated - as near as I can tell, he's assuming that organizations never get stuck in poor solutions, so they can always pick the best of a sufficiently large crowd.

Math and political correctness don't mix well.

In my experience, diversity in temperament is essential for any team to function. If you put two people who are highly excitable together, they will keep running after new ideas and feeding of each others excitement. If you put two optimists together, they will tend to gamble and play too risky. If you put a team of idea men together, you tend to have designs and nobody doing the actual work.

I believe every team should be made up of a bunch of slow and steady chaps, and a few creative people, with the creative people split into excitable and careful.

The guys that one often assumes are slow usually have really good solutions, but when the team is too fast paced, those ideas get drowned out by the people who are loud and confident of their own ability. Soon, the quiet chaps stop talking, because they know nobody will listen to them.

Diversity has nothing to do with ethnicity if you are faced with a programming task. If you were to solve a socio-cultural problem, then diversity in terms of ethnicity would probably be a good division criteria.

Very insightful. Excellent catch.

Unlikely. This is a theoretical paper, not based on any actual data. What the authors get wrong is assuming that there is no correlation between intelligence and original thinking. It is an unwarranted and frankly stupid assumption.

The claim that diversity trumps ability is defeated in nearly every real world test.

I notice that the author's advice, to hire a random sample of candidates for the position instead of the best, is not followed by any great companies that I can think of. In fact, it's 180 degrees away from the hiring practices of places like Google, etc.

It's not an either-or choice. Best-case is to hire a highly-intelligent, intellectually diverse team. Bingo Presto, life is good.

Even then, I wonder whether maximizing intellectual diversity is really the best strategy. Teams with hefty philosophical disagreements can have trouble getting on the same page enough to accomplish goals.

There may be an optimal level for intellectual diversity, but I wonder if it isn't closer to the minimum than the maximum for most tasks. It would be interesting to see real data on that; I have none.

Yes. I think you have to add social skills into the mix. Ideally maximum intellectual diversity with maximum social (negotiation, facilitation, humility, leadership) abilities, but you'd have to scale back on intellectual diversity depending on the social ability of the group.

I think that's a popular way of thinking because people like the word 'diversity,' not necessarily because it makes any sense. If you wanted to write the next Digg, say, would you hire a chemist, a programmer, and a doctor? Or would you hire three programmers? Would you hire programmers that use different languages or three who use the same language?

In practice, you get so much accidental diversity between any two intelligent individuals (we are all quite different) that its rare that you have to go looking for more. Diversity is the great good and ideal of our time, but a rather flimsy idea when you get right down to it.

Why would you want to write the next Digg? Another social news site, huh? Maybe they could actually come up with something useful. I'm sure a good chemist, programmer, and doctor could come up with something of greater utility. You need to expand your mental diversity.

If I had a good chemist, programmer and doctor, I'd use them to design rational drug design software.

Amen. There are a few groups working in this area: http://www.deshawresearch.com/ http://forums.nvidia.com/lofiversion/index.php?t60817.html

There are other types of relevant diversity, however. The authors speak directly to "functional" diversity, which they describe as "differences in how people represent problems and how they go about solving them".

In the context of wanting to write the next Digg, I would propose that a better application of their conclusions would be, e.g., to favor hiring programmers from different schools over hiring programmers who went to the same school and studied under the same professors. In this way you might get some diversity in terms of how your programmers approach solving problems versus having a group of programmers who each approach a given problem in the same way and presumably reach the same conclusions / solutions.

I agree. And it's not just types of analytical processing. How about hiring programmers who represent major different personality types, which have an innate difference in how they both perceive and respond to problems? Language knowledge, social skills, and algorithm construction is a given.

I recall seeing a photo of the Clinton cabinet (I believe), with some caption about diversity.

There were blacks, whites, hispanic, men, women, but as I looked over the faces I thought, "Lawyer, lawyer, lawyer, not sure, lawyer, ..." and wondered where, exactly, was the diversity?

Arbitrary or superficial diversity is not a good goal.

While getting on the same page is important, intellectual blind spots are a huge problem for most organizations. People who think too similarly tend to miss the same sorts of things, and it's absolutely crucial to the success of any serious project that you have diversity in terms of biases and opinions.

To make a trivial example, if you're an optimistic person and you hire all optimists, you'll ignore some potential problems. If you hire all pessimists, you'll ignore some potential long-shot opportunities you should pursue. If you hire all detail people, you'll miss out on the big picture. If you hire all big picture people, you'll miss out on the details.

People tend to hire people they feel comfortable with and get along with, which tends to mean people that think like them and agree with them. So when assembling a team, getting a diversity of opinions and biases is something you really need to take seriously. You need people who can work together and be civil and come to agreements and who like working with each other, but you really do need to have a mix of people who think differently.

I have no real data on the subject, but my mother has done organizational development work using the MBTI, so I've heard plenty of anecdotal stories around the types of dysfunctions that develop at organizations that have too much of a bias in one direction.

A better insight might be that diversity beats the median. Although even this is probably a truth with large holes in it.

If you actually read the paper, I think you will find that the authors do not, in fact, make the proposal that you are assuming.

"Should the organization hire (i) the person with the highest score, (ii) 20 people with the next 20 highest scores, or (iii) 20 people randomly selected from the applicant pool? ... In this paper, we provide conditions under which iii is better than ii."

Further, the paper is not purely theoretical, as you suppose: they then go on to provide such conditions and describe the 'computational experiment' using AI agents that they used to explore and verify their hypothesis - keeping in mind that fundamentally, their hypothesis is simply that there exists conditions under which functional diversity trumps maximized individual ability.

It's worth noting that they make a key distinction between functional diversity and identity diversity. While they do correlate the two in their abstract - with sources - their conclusion does not speak to the latter at all, instead focusing on the former and "diversity in perspective and heuristic space".

You may certainly disagree with their conclusions, but it is unfair to construct their argument as a straw man consisting solely of theoretical ideas and stupid assumptions.

"The claim that diversity trumps ability is defeated in nearly every real world test."

Completely false. C.f. The Wisdom of Crowds. The reasons why this is true aren't entirely clear, but empirical studies show that is true.

Didn't like it, personally. The book cherry-picked examples to suit its thesis from the first page. Not that there isn't Wisdom in Crowds, but J.S didn't do himself any favors by picking and choosing facts.

What would be interesting, would be to see some empirical research on a fascinating topic like this.

I agree the book itself was pretty half baked, but I suspect the thesis is correct. He's not claiming that crowds are smarter than individuals all the time, only when four specific conditions are met. He relies too much on anecdotal evidence and not enough on the academic research, but in the end I found the core of the argument convincing, if not particularly well expressed. Maybe I'm not skeptical enough, I dunno.

Crowds are wise when they go in multiple directions. When they do and someone stumbles upon something good, everyone gets to take advantage. And, if they don't agree, there's typically no penalty. And, there's usually no cost to folks wandering off.

Organizations don't work like that.

Really? Read about the New York Police "Class of '39". It was the tail end of the depression, and the New York Police had 30,000 applicants for 300 positions. They decided to choose them based on an intelligence test. As a body, the 300 people chosen attained far higher status than average police officers. Four became police chiefs, four deputy commissioners, two chiefs of personnel, one chief inspector, and one commissioner of the New York Police Department. They had far few disciplinary penalties than average.

But you are no doubt right, had NYC chosen them on the basis of "diversity" instead, things would have been much better.

You obviously haven't read the book since your example in no way relates to its argument...

How many 'original thinkers' do you need in a company made up of 50.000 people? I'm sure you're more intelligent than I am, but don't you think there are some problems I can solve better than you, based off my past experience? Diverse usually implies diverse in experience, and not in intelligence.

Short summary: The best problem solvers will think of similar solutions and so may take longer to solve the problem than a random group because the random gorp will come up with more ways of solving the problem.

Reply on news.ycombinator.com