Comment by crntaylor | Hacker News Reader

Comment by crntaylor | Hacker News Reader

Parent Full thread

The quiz that accompanies this article [1] makes an error. Question 2 asks

"A team of psychologists performed personality tests on 100 professionals, of which 30 were engineers and 70 were lawyers. Brief descriptions were written for each subject. The following is a sample of one of the resulting descriptions:

Jack is a 45-year-old man. He is married and has four children. He is generally conservative, careful, and ambitious. He shows no interest in political and social issues and spends most of his free time on his many hobbies, which include home carpentry, sailing, and mathematics.

What is the probability that Jack is one of the 30 engineers?"

The answer given is that the probability is exactly 30%, since there are 30 engineers out of 100. But this ignores all of the information available in the description! Imagine if the description said "Jack's hobbies include writing compilers, hacking assembly code, World of Warcraft, designing new programming languages and LARP." Would we still conclude that there's only a 30% chance of him being an engineer?

A correct answer to this question should apply Bayes' rule. With E representing 'Jack is an engineer', L representing 'Jack is a lawyer' and D representing the description, Bayes rule tells us

P(E|D) = P(E) P(D|E) / P(D)

P(L|D) = P(L) P(D|L) / P(D)

where P(A|B) is 'the probability of A given B' and P(A) is 'the probability of A without taking other information into account'. We can ignore the common factor of 1/P(D) and we know that P(E)=0.3 and P(L)=0.7, so the relative probabilities of Jack being an engineer or a lawyer, given the description of him, are

P(E|D) ~ 0.3 * P(D|E)

P(L|D) ~ 0.7 * P(D|L)

so the a priori probabilities of Jack being an engineer or a lawyer need to be weighted by the probability of seeing the description that we did, given that Jack is an engineer or a lawyer! If you claim that there is a 70% chance of seeing the description if Jack is an engineer but only 30% if Jack is a lawyer, then taking all the evidence into account, you should ascribe an equal probability to the two possibilities.

Basic probability fail by the editors of Vanity Fair, I think.

[1] http://www.vanityfair.com/business/features/2011/12/kahneman...

Not only do they seem to lack basic probability skills. They also cannot cite the original material correctly. Kahneman and Tversky tested something else completely. Namely, that people tend to ignore base rates when making probability estimates. From Kahneman's book:

"In one experimental condition, subjects were told that the group from which the descriptions had been drawn consisted of 70 engineers and 30 lawyers. In another condition subjects were told that the group consisted of 30 engineers and 70 lawyerss. [..] In sharp violation of Bayes' rule, the subjects in the two conditions produced essentially the same probability judgements."

There is also a blatant error in the OP article: Answer No. 2 (Linda is a bank teller and is active in the feminist movement.) is by no means "logically impossible".

EDIT: wording

'There is also a blatant error in the OP article: Answer No. 2 (Linda is a bank teller and is active in the feminist movement.) is by no means "logically impossible".'

They meant that it's logically impossible for the probability of A AND B to be higher than the probability of A, since A AND B is a subset of A. She could be feminist, but the probability of being feminist is lower than the 100% probability that she's either a feminist or not.

I am well aware of the point they were trying to make. Or, more precisely, the point Kahneman et al. were trying to make. What I am concerned about is that they, that is, the editors / writers of Vanity Fair, do not understand their mistake. This is somewhat likely given the error they made described by the grandparent comment.

Overall, I find it rather ironic to find such gross errors in an article that sets out to educate the reader about "human error".

I'm going to copy/paste the relevant section:

Then they went around asking people the same question:

Which alternative is more probable?

(1) Linda is a bank teller.

(2) Linda is a bank teller and is active in the feminist movement.

The vast majority—roughly 85 percent—of the people they asked opted for No. 2, even though No. 2 is logically impossible. (If No. 2 is true, so is No. 1.)

They say: "If No. 2 is true, so is No. 1.", but they omit that the opposite does not hold. Perhaps they thought it was obvious enough.

I think they get it and just left that second part out. They may have worded it poorly (better would be "it is logically impossible that #2 could be the correct answer"), but it was very clear to me when reading the article that the editors meant exactly what you describe.

Of course, the real study is not encompassed by this question. The real study checked if the answer given was different if the prior proportion of lawyers was 30% with 70% engineers or 70% lawyers and 30% engineers. The estimates people gave did not vary based on prior probability.

Now that I think about this one, I think maybe the probability really is 30%. They don't say how they selected the lawyers and engineers. Maybe they selected patent lawyers (required to have engineering degrees) and all 70 members of the Society of Women Locomotive Engineers. Maybe they selected people at random. We don't know. Since we don't know the priors, we can't apply any reasoning about any characteristics of Jack.

In fact, they didn't say how they selected Jack either. Maybe they picked him out just to humiliate quiz takers who don't know the PI is a sadist.

This is a classic case of abusing the student by proving trick questions without essential information.

Wow. I was going to disagree with you... then I read the quiz and note. It's worse than a mere error, the quiz writer is willfully throwing away knowledge about the world.

They didn't give conditional probabilities but of course we can, and should, use what we know about the world to make guesses at P(E|'likes math'), or at least whether that's greater than P(L|'likes math'). It surely is and you can do something similar with carpentry and an apolitical nature.

It's true that even with the assumptions I've given above you can't come up with a precise answer - none of the choices are any good - you can certainly say that "...the correct response being precisely 30 percent" is thoroughly wrong.

Reply on news.ycombinator.com