Overall, I think YouTube was on the ball by switching to a "Thumbs up, Thumbs down" rating system. "Out of five star" rating systems are always sabotaged by the fact that most people think in absolutes: Either something is the bee's knees or it's fried shit on a biscuit. Given that most people vote this way, up/down vote systems are better at giving an honest impression of how people feel about something.
I think Barbara Ehrenreich mentioned that in Nickel and Dimed. At Walmart, one of the hiring surveys included some question pitting personal considerations or ethics against loyalty to management, and the only correct answer was absolute loyalty to management. When she debated the point, she was not offered a job.
My boss was never happy with the speed at which I worked. Eventually she told me I was too smart for the job and should aim higher. In a vaguely similar way, I don't think Wal-mart is looking for their lowest-level employees to make a lot of independent ethical decisions on the job. If you're going to be hiring unskilled labor, you might not want them to exercise a lot of discretion, on the theory that they're at least as likely as not to screw it up.
Specificity is one think I really look for. Those who say essentially nothing more than "It's wonderful" or "It sucks" are noise. Those that say, "It's great, but I'm subtracting one star because the control console is a pain because of X, W and Y" are golden ... especially when people add comments on workarounds....
It's kind of funny, in that this "impersonal" mail order outfit very often manages to provide a better shopping experience than brick and mortar stores for those of us willing to buy things sight unseen. Heck, it's not uncommon for me to check Amazon.com reviews before buying something from one of those.
OK, granted, I've been an Amazon.com customer since 1997 or so, but to me a 3.5 star rating is like writing BEWARE! in flaming letters across the sky. I never buy something that poorly rated before thoroughly investigating what's behind it. E.g. Kindle version's content is messed up I discount since I don't play in that ecosystem except for freebies.
While there are plenty of mostly bi-modal distributions (5 or 1 stars), for lots of things there are carefully chosen ratings in-between, often telling you why it lost a star or 2. Even ones edited some time after with additional commentary and often an adjusted score.
So I'd say it gets the specific job done, which is quite different than YouTube's where the cost of a wrong choice is very very small. I find the reviews invaluable, to the point where there's a lot of stuff I won't buy unless it has enough quality reviews for me to make an informed judgement.
For example, one common shape looks like this:
5 stars: |||||||||||
4 stars: ||||||||
3 stars: ||||
2 stars: ||
1 star: |||||||
If the number of star ratings follows a more normal curve like the following, then I'm more confident in the quality of the product, even if both have an average 4 star rating:
5 stars: |||||||
4 stars: ||||||||
3 stars: ||||
2 stars: ||
1 star: |You could study Amazon, App Store ratings, etc. Cool insight!
All that said, there is information to be harvested in some of these reviews.
It would be interesting to see a trend line of change in sentiment about a products over time, using existing data.
5 stars: ||||||||||||
4 stars: ||
3 stars: ||||
2 stars: |||||
1 star: ||||||||There was a fascinating post to the OKCupid blog discussing the distribution of attractiveness ratings (also on a 1-5 scale).
Ratings of women by men looked like this:
5: ||||
4: ||||||||
3: ||||||||||
2: ||||||||
1: ||||
5: ||
4: ||
3: |||
2: |||||||
1: ||||||||||||||||||||
Obvious corrections:
Scale is 0-5, not 1-5
Curve for women is not as steep as depicted in my comment. However, the modal rating is 1, followed by 0. In the words of the post, "women rate an incredible 80% of guys as worse-looking than medium. Very harsh."
Of course you have to look at the ratings distribution and not just the average. The average tells you almost nothing.