Mathematicians and the Selection Task (2004) | Hacker News Reader

Mathematicians and the Selection Task (2004)

eric.ed.gov

48 pointstzs26 comments

An interesting thing about the Wason selection task is that people do a lot better when given a task that requires the exact same reasoning but involves social situations.

For example if the cards have on one side either a picture of a mug of beer or a picture of a can of soda, and the other side have a number representing the age of a person drinking that drink, and the rule they are supposed to be checking is that if someone is drinking beer they have to be at least 21 then 75% of people correctly figure out that they need to check the other side of the cards showing beer and the cards showing an age under 21.

Here's Wikipedia's article on the Wason selection task [1].

[1] https://en.wikipedia.org/wiki/Wason_selection_task

I'm not sure it's true to say that that task requires the exact same reasoning - There are various logically equivalent ways of phrasing the rule such as "No card has a D on one side and a number other than 3 on the other" which make the problem easier.

Since the rule "you can't drink alcohol if you're underage" is one people are familiar with, they aren't being asked to make the same logical deduction they do in the letters and numbers question. I'd go further and speculate that they aren't all reading the question carefully - if you replaced the rule by "if someone is over 21 they are drinking beer", how many people would get it wrong?

It seems to me that the selection task is tricky because it concerns interpretation of language. "Every card that has a D on one side has a 3 on the other" makes a claim: that there is a directional dependency "D => 3" but it makes no claim that "3 => D". However the absence of the latter claim is not stated explicitly, it is supposed to be inferred from the original statement. The English language seems to lack a way to encode unambiguously the "A => B" relationship. So it should not be surprising that students used to looking out for language pitfalls when checking proofs also happen to be the students who do better on this task.

> The English language seems to lack a way to encode unambiguously the "A => B"

It doesn’t: “If A then B” encodes it unambiguously.

It’s just that as you said, many people don’t think hard about the difference between this and similar-but-different concepts like “B only if A”.

It’s not the language itself, it’s the way people use the language and think about what it says (or in this case, don’t).

> It doesn’t: “If A then B” encodes it unambiguously.

No, actually it doesn't

> Some authors have argued that participants do not read "if... then..." as the material conditional, since the natural language conditional is not the material conditional.

> It’s just that as you said, many people don’t think hard about the difference between this and similar-but-different concepts like “B only if A”.

While a nice simplistic answer it's likely not what's going on here. There is more here than "People just aren't good at thinking".

If your statement were true, then you'd be forced to say that the following statement is also obviously true:

"If the Nazis won World War II, then everybody would be happy"

The fact that you can rightfully say that sentence is false, means that your comment above about implication and "if" statements is wrong.[1] Language is more complex then you're giving it credit.

[1] - https://en.m.wikipedia.org/wiki/Paradoxes_of_material_implic...

For what it's worth, I think that the problem lies in the solution feeling "obvious" and people being a little lazy - I don't think the problem lies with inability.

I also think that if A then B is unambiguous, the counter that languages are different doesn't really fit with what I think I observe in the wild. For the full house, I also fail to see how that means I must accept that your example statement is obviously true.

I just failed the test. :) The only thing that this teaches me is to pay more attention before attempting to solve a problem.

One must ask if the investigators of this study selected the correct faculty for examination. (I kid, this is embarrassing)

As a mathematician, I was primed with the knowledge that a large fraction of a mathematics department failed this test. I looked it up on Wikipedia, didn't spoil the answer, and thought damn hard before unfolding the "solution" section. I was relieved to see my answer therein. I do wonder if the students and staff were primed to think about this as a logic puzzle, or if they simply went with a gut answer. Because in my experience, that makes loads of difference in how people of all stripes, mathematicians included, respond to challenging questions.

My gut response was to flip an extra card, for what that's worth. Secondary consideration took a couple of seconds, and I spent another thirty convincing myself that I was correct.

I think we're looking at this wrong. I feel like this test is designed to investigate social biases not test for logical skills and if these people are failing it, it's not so much of a failure in their understanding of logic but rather a procedural impact of the way the question is framed, which is probably precisely why "reframing it in a social context" changes their result populations. I think this test is extremely sensitive to how you pose the question.

Are we trying to test if the candidate can solve the logic problem, or are we trying to test how they handle an /intentionally-confusing/ situation and what (psychological) biases they jump to with their solution?

If it's a test of their logic capabilities, then it seems like the numbers are artificially low, so maybe not so embarrassing as you say... Reason being, I think there are several confounding variables included in the results they'd need to control for if that was the point.

An obvious one, if we were testing logic directly, then I wonder if they allowed the participants submission to "show their work" rather than just which final cards they chose. Doing so would eliminate the "carelessness" confounder in the result where they didn't thoroughly think through all of the logical cases of the cards or where they accidentally included an incorrect card but understood the nature of the required solution, ie. if they knew they needed to disprove rather than confirm but accidentally included a useless card for disproving, they still understood how to solve the problem and thus the logic. What percentage of their results fall into that bucket?

There's also other confounding factors that are set up to "confuse" the participant here that could be removed if we wanted to truly test their /logical skills/ and /not/ some psychological/sociological property. For example, the question merely says: "test that if a card shows an even number". In English, "if" can mean both the inclusive or exclusive OR depending on context - it's needlessly vague, and additionally, I posit that in English, given the common usage of the phrase "test ... if", the phrase is /leading/ the participant to look for /positive confirmation of the rule/ rather than the negative. You can of course derive that the negative test is needed by studying the cards but why try to mislead them outright? Why not say "choose the set of cards that you'd need to flip to prove the rule is false"? This clearly demonstrates the task and doesn't send them on a goose chase.

There's other things too. It doesn't mention if these cards are from a global set of cards or the rule is only meant to be proven on the 4 cards presented. It implies the latter but if you start thinking about "confirming if the rule is true for all cards", it sends you down another useless logical rabbithole, yet, /cards normally come from a deck in real life/ and it is natural to expect there are more cards. Maybe if they wanted to be exact we shouldn't be using cards at all but rather wooden blocks or something.

And I'm sure there are more "biases" that I'm not catching here. If your goal is to test people's likelihood of affected by certain biases psychologically, then all's well and good with the test, go right ahead. But if you're going to present the poor results as some sort of indicator of an population's skill at logic, maybe not the best test without some better testing procedures, imo.

This result reminds me of a paper I read last week via Andrew Gelman's blog [1]. It's a very thorough review of the, so called, bat and ball problem and is an up to date summary of something brought to many people's attention via Kahneman's Thinking Fast and Slow. As other commenters have suggested, the most reasonable explanation for the mathematicians to get this problem wrong is something more like carelessness than a lack of logical reasoning ability.

[1] https://statmodeling.stat.columbia.edu/2024/04/21/now-heres-...

"Four cards were placed on a table: [D K 3 7] The participants were given the following instructions: Here is a rule: “every card that has a D on one side has a 3 on the other.” Your task is to select all those cards, but only those cards, which you would have to turn over in order to discover whether or not the rule has been violated. The correct answer is to pick the D card and the 7 card,..."

Wait a minute, this is wrong! You have to check the K card also, since it could have a D on the hidden side. Only if you were given the instruction, which is done later, that all cards have a number and a letter on one side, you know that the K card must have a letter on the hidden side.

I think it's given that there is a letter on one side and a number on the other side if every card and implicitly, that that rule hasn't been broken, otherwise you have to check every card to see that there's no emoji there

We have to make sure, that if there is a "D" on one side, there must be a "3" on the other side. If we see a "3", we are done, otherwise if there is not a "3" we have to check that there is no "D" on the hidden side. Hence we have to check exactly all cards facing a "3".

No, because the framing of the problem includes a statement that if there is a letter on one side then there is a number on the other and vice versa. The K can't have a D on the other side unless they're lying in the problem statement (which would defeat the purpose of the experiment, so why would they and why would you assume they are?).

I now how the experiment was conducted. That wasn't the question either. Read carefully what i have cited. There is nothing about what you have said. If you specify something, you ought to be very precise :-)

> Participating subjects were shown a selection of cards, each of which had a letter on one side and a number on the other.

You skipped that part just above your quote. There is no ambiguity. In both this experiment and the original experiment they were given that information before their selection.

Quoted sentence does not support your claim.

beyondCritics was attempting to be pedantic, and failing due to not reading, that the problem in this linked article was not presented with the context that the cards were always letter/number pairs and never letter/letter pairs (in which case you'd need to check the K card). Just a few sentences earlier in the article from what they quoted, though, it presents what I quoted, that the cards are letter/number pairs.

To support my claim though:

From the experiment in the linked article:

> Four cards are placed on a table in front of you. Each card has a letter on one side and a number on the other.

That statement is included in the text provided to participants.

From the original experiment:

> The subjects were told that cards with letters on their front had numbers on their back and vice versa.

Subjects were tested only after being given this information. Satisfied?

https://web.mit.edu/curhan/www/docs/Articles/biases/20_Quart...

As an math educator, I think there's a huge flaw in this study. The investigators failed to follow up to see why the mistake was made. They leap to assuming the player is trying to "deny the antecedent", but I think there's a much simpler explanation: the players aren't reading the instructions carefully.

There's two reading errors I would expect someone to make given this experiment:

1. The instructions that each card has exactly one letter and exactly one number are before the big cards. I bet many players just skipped that instruction.

2. Mistaking the P→Q as P←→Q smacks more of a reading comprehension error than a logical error.

Disclosure: I made both mistakes. (-:

> less than a third of students and less than half of staff gave the correct answer.

This is incredibly troubling. If universities cannot produce people that can consistently get these kinds of problems right, what the hell are they even good for?

Not necessarily more troubling than being tricked by an optical illusion. Perhaps this problem is more like a logical illusion because of the presentation/wording.

I think the fact is that although predicate logic is a foundation of mathematics, it is not what mathematicians spend the majority of their time thinking deeply about. You might use English every day of your life, but still struggle to explain what a transitive verb is, or a gerund.

It would be interesting to run this experiment on CS or EE students. I hope they would do a little better…

All people would do better if they are warned first it's a tricky question.

Also, training in this kind of problems will help.

I think you are presuming that the participants who failed were unable to solve the underlying logic problem, when it is entirely possible that they (eg) misread part of the problem setup.

(Likewise the paper seems to infer a difference in logical thinking rather than considering a difference in processing/interpreting the problem.)

Maybe, but understanding a problem is half its solution. Misread and couldn’t solve correctly are the same thing in problem solving.

Reply on news.ycombinator.com