You are simply mistaken in your interpretation of the technical term "unbiased estimator". This has a specific meaning in statistics, and is required for the convergence property you specified earlier. From wikipedia [0]
"In statistics, the bias (or bias function) of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated."
In lay terms, this means that the estimator process "ask lots of people and average the result" is unbiased only if all the too-high errors, in aggregate, cancel out the too-low errors.