Comment by indubitable | Hacker News Reader

Comment by indubitable | Hacker News Reader

Parent Full thread

You'd think so, but that's not correct. This is not a straight forward result of probability with a filter of complexity. It works regardless of bias, though obviously if everybody was biased in the exact same way then that would cause things to break down - which is perhaps the reason that the coordination results in a worse result than independent averages.

For another example of it consider things like the television show, 'Who wants to be a millionaire?' It's a quiz show where one of the choices is for the participant to ask the audience. And the audience tends to do absurdly well on even the most esoteric questions, though independently they are certainly far from trivia experts. But very few are randomly guessing - their own experiences and biases leads them to entirely different conclusions. Yet somehow it produces the correct result time and again.

It's a strange phenomena and one that has to be constantly guarded against in anything involving sampling of people. This is a text book example of a study that gets destroyed by it.

>>> As the number of people independently asked approaches infinity, the error approaches 0.

>> Only if people are an unbiased estimator!!

> You'd think so, but that's not correct.

Either you're misinterpreting the technical term "unbiased estimator" here, or you are aware of some research that I would like to read.

In context, "unbiased" means that if you pick people at random and ask for their estimates, then on average the too-high estimates cancel out the too-low estimates (i.e. there is not a bias in one direction or another).

But people as a whole have poor understanding of many things. One common one that appears in social science research and is often replicated is that people grossly overestimate the size of the homosexual population in the US (the "wisdom of the crowds" often estimates it around 20% whereas best available polling data suggests 3-5%). Here's just one source for this phenomenon

http://news.gallup.com/poll/183383/americans-greatly-overest...

"Wisdom of the crowds" is occasionally reliable, but it should not be assumed to be reliable for any particular problem without verification. It often fails terribly even on problems that are not very esoteric.

Edit: changed phrasing of final paragraph

As mentioned, there is a difference between bias and uniform bias. In the US the media, politics, social media, and even miseducation (e.g. in my deviance class we focused on Kinsey's 10%, yet oddly enough never contrasted that against contemporary results) have heavily and uniformly biased the population on sexuality leading people to vastly overestimate the number of homo/bi/trans individuals.

Where it works phenomenally well is in areas where biases have not been directly instilled into people. This does not mean people are unbiased, however. Again the knowledge of trivia is a good example since while the crowds can generally do phenomenally well even at very esoteric questions where biases would lead them to individually come to very different conclusions, yet they will invariably fail to answer ostensibly trivial questions like 'What is the capital of Australia?' You'll get Sydney, it's not. You'd likely get a similar result for things like the capital of Pennsylvania.

In a way I view the wisdom of the masses as analogous to machine learning systems. They do an oddly good job of providing extremely precise answers to a wide array of questions even when trained with models that do not directly represent the 'questions'. Yet you can also break the systems, at times comically, with certain types of queries designed to do precisely that.

And as was the case with machine learning for quite some time, I think people remain reluctant to utilize it due to the black box nature of it. The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse.

> As mentioned, there is a difference between bias and uniform bias.

You are simply mistaken in your interpretation of the technical term "unbiased estimator". This has a specific meaning in statistics, and is required for the convergence property you specified earlier. From wikipedia [0]

"In statistics, the bias (or bias function) of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated."

In lay terms, this means that the estimator process "ask lots of people and average the result" is unbiased only if all the too-high errors, in aggregate, cancel out the too-low errors.

[0] https://en.wikipedia.org/wiki/Bias_of_an_estimator

I'm not sure if we're now going in circles or if you failed to read what I just wrote. Repeating it:

"The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse. ... "

I'm not sure if you even realize all the assumptions you're making. You are assuming, for instance, that 'guesses' are regularly distributed. That assumption may be correct in some cases - I expect in many it is not. Alternatively there is the possibility is for you to claim that you're referring not to the individuals in question as the estimators, but the entire group. In that case you've spent a lot of time saying nothing as it boils down to "people are only correct if they're correct."

Sorry but you're mathematically wrong. The "wisdom of the crowds" does not work on things were everybody is wrong in the same way. For example if you ask a lot of people to estimate income inequality you get the wrong answer because everybody underestimates it.

https://www.scientificamerican.com/article/economic-inequali...

The "wisdom of the crowds" is pseudo-nonsense.

Reply on news.ycombinator.com