Why 50% of MIT students get the bat and ball problem wrong | Hacker News Reader

Why 50% of MIT students get the bat and ball problem wrong

mannhowie.com

These are the three questions of the Cognitive Reflection Test:

(1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? __ cents

(2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? __ minutes

(3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? __ days

Here's another question for the Cognitive Reflection Test:

In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton got a question wrong on a Cognitive Reflection Test. What percentage of MIT students got the question wrong?

What is the answer?

Think its meant to be tongue in cheek. Read the OP link, and you'll see where it came from.

You cannot answer from the data in the question ;)

Here's ChatGPT's responses. It's fascinating that it provides the same incorrect answers as people... I'm not sure how to interpret this from the robot.

(1) "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total."

(2) "If 5 machines can make 5 widgets in 5 minutes, then each machine can make 1 widget in 5 minutes / 5 machines = 1 minute. Therefore, 100 machines can make 100 widgets in 100 minutes."

(3) "If it takes 48 days for the patch of lily pads to cover the entire lake and the patch doubles in size every day, it would take half that time, or 24 days, for the patch to cover half of the lake."

This post title is heavily editorialized. The article title is "Cognitive Reflection Test - Examples of Reacting vs Checking", and doesn't really attempt to give a canonical answer the question of why many people fail the CRT. It discusses some possible reasons (mainly Kahnemans two modes of thinking) and proposes a 3-step strategy to to avoid falling into the trap of going with ones initial, gut feeling.

The actual reason for the high proportion of failures doesn't seem that complicated, personally I'd label it as "falling for trick questions" and is something everyone is likely to do if such questions come out of the blue, especially if under some time constraint. Our minds tend to be as economical as possible.

yes exactly, i "failed" that test myself but then i thought, why didn't i do better? Because the cost of failing in this case is absolutely nothing, and the brain automatically optimizes for speed with the first random answer.

The 3 questions quoted in another comment are not trick questions and they obviously did not come out of the blue for anyone taking the test.

They are simple reasoning questions. IMHO, people tend to "fall for them" not because they are trick questions but because people don't really reason carefully.

The standard advice I have always received is to read the actual question carefully, the whole question and nothing but the question.

This might catch a random member of the public but you would think university students are at their peak of "reading test questions skills".

Here I think what the questions do is that they may cause our brain to very quickly come up with a shortcut and wrong answer, which should immediately be discarded if you were indeed carefully reading the question, i.e. paying attention (so maybe indeed something like a conflict between 'fast and slow thinking').

I really tried to analyze my thought process before/during/after seeing the problem, trying to solve it, and then seeing the solution.

It seems to me that a lot of our thought process isn't under fully conscious control. We're pattern-matching machines, and math and logic don't come naturally to us. To apply math or logic, you have to train your brain to look at a problem without any shortcuts. Even then, you sometimes need to look at a problem from another angle. For instance, I think that people who read this problem out loud, slowly, may spot the trick more easily than those who just try to reason about it silently in their minds.

The problem for me was the difference between a constant and a variable.

"Costs a dollar more" is not saying that the bat cost $1 as a constant. This was a variable, relative price compared to the ball.

Failure to understand the mutability--or lack thereof--of the symbols in the system is a source of deep woe.

IMHO, the point is you need to carefully read and analyse the question. Don't let your "fast brain" trick you.

Regarding fast and slow: "It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted."

https://replicationindex.com/2020/12/30/a-meta-scientific-pe...

I wonder if the 50% test has been replicated...

I asked ChatGPT this question. Here's its first response:

Q: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?"

A: "The ball costs $0.10. The bat costs $1.00 more than the ball, so it costs $1.00 + $0.10 = $1.10 in total."

Fascinating that it gets it wrong exactly the same way that a human gets it wrong. I kept trying a few times, and eventually could get a correct answer out of ChatGPT. But most of the time it gives the wrong answer.

What about other students?

Specifying MIT et al. makes it look like it’s a specific flaw in stereotypically ‘smart’ people, rather than the general population.

Curious if this is more generally tested and known, or what biases are in place in specific populations. Eg. Bias to immediately trust first thought/gut, vs being taught to be more thoughtful and cautious.

Why is there no sales tax? If you're going to do story questions, make them realistic, or else they're just confusing garbage.

For those prices it was clearly something like a garage sale purchase, with no need to pay taxes.

Cheapest bat&ball at walmart.com is $10.

Could be a clearance sale, used sporting goods store, in the past, in another country, etc. It's just not a good question for anything but tricking or frustrating people.

While true, my scenario is not unrealistic.

Also, if you include "in another country" then other countries include taxes in the item price.

Yes, that's my point, there's no way to know without making assumptions. Making assumptions about important details is sloppy and irresponsible, not what should be promoted by putting it on a test. They could have easily used a the same setup with weight or speed or anything else but money and it would have been a fine question.

Life requires making assumptions.

Understanding the specific tax requirements for this exchange of goods is not an important detail.

Your overthinking the problem. As posed, the question doesn't even get to the point where the bat and ball have been purchased, which is where tax needs to be paid.

If I'm shopping and someone asks me "I'm sorry, I left my reading glasses at home. Could you tell me the price of the pimentos", then I might say "$2.99 for the jar."

I won't say "$2.99 for the jar, plus applicable sales tax", and neither would the vast majority of people.

That’s life though. It’s reasonable to ask clarifying questions on a test. Word problems are notoriously tricky and one shouldn’t fail because they simply made a wrong assumption.

You are assuming that's the main reason people get the question wrong.

IMO your assumption seems very unlikely.

Your conjecture that taxation affects interpretation can be tested by international comparisons, between the US (where shelf prices do not include tax) and another country where shelf prices include tax.

A Google Scholar shows this test has been carried out in many other countries, including those where prices include tax. The few papers I skimmed (like https://link.springer.com/article/10.3758/s13428-018-1029-4 ) make no mention of your conjectured effect, suggesting my opinion is widely held.

Yes, it's just about impossible to make a word problem which cannot be over-thought, even one based on a real-world example.

Sales tax was honestly the first thing I thought of as I read the question, because I've encountered such trick questions in the past in math classes and on 'cognitive tests'. I'm also really boggled on why you think training future engineers to make assumptions about numbers isn't really bad.

I never made such a blanket statement.

I said that the question as posed is not "confusing garbage".

Based on the price, I assumed it was a garage sale and not really taxable. But, ok, let's say it's a fire sale and is taxable.

How would sales tax affect the calculation at all?

If it does, how would you write the question in a way to assuage your concerns about taxation?

Does your text cover all the relevant details? Like, is the entity making the purchase subject to sales tax? Or, does the purchase take place in West Virginia when this purchase might be covered under the Annual Back to School tax holiday?

Reply on news.ycombinator.com