Ignoring the Wisdom of Crowds | Hacker News Reader

Ignoring the Wisdom of Crowds

blog.asmartbear.com

49 pointsjoeyespo19 comments

I stopped taking this seriously when they compared the accuracy of the "ask the audience" and "phone a friend" bits on Who Wants to be a Millionaire without considering that everyone used the ask the audience lifeline on an early question and saved the phone a friend for later questions.

Without controlling for when they are used the comparison of accuracy is literally meaningless.

The same stats show that, rightly or wrongly, the contestants didn't have as much faith in the audience for later stage, much harder questions.

I guess I should give the benefit of the doubt and assume the writer didn't notice this and the error is accidental but that's only slightly better than just ignoring it because you wanted a sexy anecdote to support your point.

You right that the two lifelines are used at differing stages in the game. I wonder though, what the accuracy rates look like after controlling for what stage of the game you are in. Maybe people SHOULD save their ask the audience for later in the game.

Another variable is the subject matter of the question. Anecdotally, it feels that the audience almost invariably nails the answer by a wide margin on pop culture questions. On questions of a scientific or technical nature, the audience opinion is much less clear and more often actually wrong. Phone those questions to a domain knowledge expert.

About the worst thing you can do, but contestants love doing this, is to burn the valuable audience lifeline early on a question when you already have a good sense of the answer but just want to "make sure". What a waste! Save that for a question when you have no clue, which WILL happen before you reach the million level.

Something else they often do is say something like "I'm thinking it's B, but I'm not sure so I'll ask the audience". Well guess what the audience guesses after hearing that?

My intuition is that the accuracy would drop a fair bit in later stages and anecdotes support this too.

My intuition is that for ask the audience to be useful the number of people who know the answer has to be well above the noise or you don't get any wisdom from the crowd.

So that means that enough people have to know the right answer AND the wrong answers should be distributed as evenly as possible.

People don't often answer multiple choice trivia questions wrong in an evenly distributed way so this makes sense to me.

It would be cool to see some actual data on this though, I thought I remembered seeing something back when the show was huge but it's a vague memory.

If you stopped read at this point you missed the main idea of the essay. It is excellent.

Didn't stop reading, just stopped taking it seriously.

I agree with the conclusion that the wisdom of the crowd doesn't apply to creative endeavors, and the reasoning why. The framing of the vote as a veto was stating the idea in a nice way.

But that means that all it is is a conclusion I already agree with prefaced by bad logic.

His further reading list includes Surowiecki's The Wisdom of Crowds, but his post doesn't look like he read it himself. Surowiecki has a long list of requirements for the "wisdom of crowds" to actually be helpful, for one example the individual judgments need to be independent, that the post ignores. Without the restrictions you get different failure modes, like bubbles, runs, and traffic jams.

Or election mistakes. Democracy assumes people's choice is independent, though it is not. We are still very exposed to manipulation.

The essay was intended to be a light-hearted and readable intro to the concept, not an exhaustive definition that takes an entire book!

Almost everyone has at least heard the phrase "the wisdom of crowds" by now. The most important part of the idea, and the one most people are not familiar with, are the (fairly severe) limits on its usefulness. I wrote a review of the book for Amazon, and one thing I pointed out is that it was a much better balanced treatment of the subject than I had expected from all the hype about the idea that I had previously read. The hype is already all over the place, and has been for years, there is no need to add to it.

As programmers we should be well aware that our customers' "needs" are bounded by what they a) know and b) believe is possible. Guiding them to technological enlightenment can and does provide rewards (financial and otherwise).

Are you essentially saying that if you listen to your customers you'll build a faster horse, instead of a car?

No ... I'm saying that sometimes we are evangelists and can make them realize they want a flying car.

He isn't being very fair: where the goal at first was to be as right as possible, it later became "to piss off as few of the 300 guests as possible". I'm not sure what exact procedure he had in mind for businesses, but in a situation where "votes eliminate the interesting edges, leaving only the boring residue that no one hated enough to vote off the island" applies, you're just not talking about the wisdom of crowds anymore, this wasn't possible in his earlier examples either. An important factor to the wisdom of crowds, and a key difference between his earlier examples and his later examples, is that people vote independently, without even talking about what they're going to vote.

For demonstrational purposes, I will now take his holiday meal situation and apply actual wisdom of crowds to them. This isn't a very usual situation, but this is about properly applying the wisdom of crowds.

Instead of people stating what they dislike, people may nominate a meal, and afterwards, everyone votes for the meal they want to go with. Like in the Wants to be a Millionaire examples, people cannot talk in this process - they cannot state what they dislike. Instead, the fact that some people can't take spicy food will show in the results, but beside that, every individual will pick a meal that isn't boring, and the end result will be just that. "Allergic to garlic" and "doesn’t eat anything green" are rather specific complaints and were not taken into account, so let's count those 3 people as unsatisfied, but the success here is that most people will in fact be satisfied with the meal.

You can't use your technique for meals because everyone will suggest a different meal, and then you have no basis for a decision. That technique works when you're voting on one variable which has one CORRECT value which you're trying to seek. Creative work is not that.

The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks.

The advantage of focus groups is, you can identify themes which resonate. For example, the group might say "integrated" sounds good. That doesn't mean you want an integrated product, it just means that you need a sticker which says "integrated", and some justification (We integrate with Facebook, via our "Like" buttons!)

>The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks.

Experienced group moderators are aware of this take steps to mitigate and reflect in their notes.

There are other forms of market research beyond focus groups. Aggregated quantitative and qualitative one-on-ones also fit the OP's bill of "crowd wisdom"

This essey is weak, I mean author is comparing apples to oranges. First problem with jellies is Fermi problem, it is approximation. If you have more people giving they answers mean of approximation will be closer to actual value. But second part is about concrete thing it is not estimating or guessing. First part is totally irrelevant. Second part could be used alone. But you just know that you can't please everyone so it is common sense I think. Like going for trip with people who are working in different jobs on different schedules, somebody will have to be left out.

Reply on news.ycombinator.com