The candy weighing demonstration, or, the unwisdom of crowds
andrewgelman.com
andrewgelman.com
Maybe it's because I recently did some reading about magicians, but if I were one of the students I would be thinking that he could have any number of hidden envelopes with different predictions, and he just chose the one that ended up being correct. Of course, I'm deliberately missing the point of the story.
Yes, but it wouldn't be so impressive if he were to choose the envelope which said "your guesses were all wrong, and normally distributed about the actual value".
What is demonstrated is when you give the students an algorithm for a biased estimator, the estimate they get is biased. This is good; empirical demonstration is useful... but it isn't the wisdom/unwisdom of crowds, really.
edit: good responses! Unfortunately I don't have enough time right now to properly clarify how/why I'm looking at it this way.
He is then simply showing that for certain populations such approaches won't necessarily give good expected results, and so one should always exercise caution - whether the population is sweets or people's guesses at the weight of a cow is essentially immaterial.
This is a perfectly good demonstration of bias in statistical estimators. It just doesn't have anything to do with crowds, their wisdom, or their unwisdom.
The wisdom of crowds effect is not a 'strong result', and it can be hard to work out whether it will apply in any given situation, and so it is prudent to exercise some caution, and to teach people why said caution is warranted, which is the point of his little game.
The 'wisdom of crowds estimate' (take 5 samples and average) isn't a biased estimator, because averaged across all realisations it would produce a correct result. The bias creeps in mostly because of careless assumptions on the part of the samplers.
You said "that misses the whole point of why the "wisdom of crowds" actually works when it does" - I guess all he's saying is that it doesn't always work, which I presume you agree with, and he's giving an example of why. To flip your argument around: saying that the wisdom of crowds works when it works is simply tautological.
edit: just to clarify, the "crowd" here is the group of sweets, not the participants. The participants are multiple realisations of the estimator, at least that is my interpretation (because if not, then I agree with you - the participants aren't a "crowd" in that sense).
But they are all using the same estimation method from this biased information, and they are all (by construction) creating the same bias, and that is very much not crowd-like behavior.
Applying this as an example of the problem with "wisdom of crowds" seems to really miss the point to me, because you created a situation where they didn't behave like a crowd.
Whilst I still think you can interpret his experiment as showing something about how and why WoC can go wrong, essentially along the lines I previously argued, it is not obvious, and there is some subtlety in the interpretation.
Your original phrasing "the "wisdom of crowds" part of the title is a bit unfortunate" seems most appropriate; indeed the word "crowd" doesn't appear in the rest of the piece.
I agree the estimator is biased, but in good part because of how people pick 5 from 100.
We are all biased in some ways. If a group shares the same biases, then a wisdom-of-crowds approach will yield a wrong answer in which everybody is confident. I think it's a fine lesson for people trying to work in this fashion.
They knew their goal was to estimate the bag, and they were allowed to pick any five candies they wanted. If somebody picked in proportion to frequency, they'd be fine.
We were told to measure exactly 25 ml of both water and alcohol, mix them, then report the volume. It was a competition to see which team got the closest to 50 ml...
So we all reported our values on the board at the front for comparison.
At the end, the team closest to 50 was declared "A Liar" because alcohol and water do not mix to the same volume as their constituent parts, thus the final volume should have been closer to 46ml and the team closest to 46 was awarded as the winner of the competition.
Blew my 15 year old mind and I never biased my answers towards my bias again.
As you pass the 1.5 kg bag to the next group, it's easy to see that that it's not 3-4 kilos as you have just guessed.
I guess that still doesn't prevent someone from weighing all of the candies, one small batch at a time, and adding up the results. But hopefully in a classroom it would be obvious if someone was trying to cheat that way. I suspect that asking the students to work in pairs helps to discourage those kinds of shenanigans, too.
http://phenomena.nationalgeographic.com/2013/01/31/the-real-...
This post is more "ask a bad question, get a bad answer".
That's how you increase fruit content in your müsli - shake the bag, scoop from the top and leave oats to others :)
So nobody in the class just waits until you're not looking and weighs the whole bag with the scale?
Or simply a scale incapable of measuring a weight much larger than the heaviest piece of candy.
On similar ground, this reminds me of Deming's red bead demonstration, relating statistics to corporations and management practices. Best explained dynamically: https://www.youtube.com/watch?v=JeWTD-0BRS4 (delightfully, this is posted by the Mayo Clinic).
The first reaction I think any class I was in would have to this demonstration would be to figure out how we're being cheated.
Given a bag with a random sampling of candy and being told to 'pick 5 pieces' I doubt I would choose 5 of the same large candy bars.
It seems highly surprising that 100% of the time this is done you don't have a single pair of students reaching just a bit further into that bag.
It doesn't even seem to be a particularly impressive demonstration.
I imagine the use of a bag rather than a Jar as per the usual school fair game could make spotting these harder though (unless they're allowed to pick up the bag).