Hacker News Reader

Comment by idProQuo | Hacker News Reader

This has always bothered me, both on Amazon and the iTunes app store. No lie, when I published my first app, one of the comments was "I'd rate this zero stars if I could." and my first thought was actually "wait, why can't you?"

Overall, I think YouTube was on the ball by switching to a "Thumbs up, Thumbs down" rating system. "Out of five star" rating systems are always sabotaged by the fact that most people think in absolutes: Either something is the bee's knees or it's fried shit on a biscuit. Given that most people vote this way, up/down vote systems are better at giving an honest impression of how people feel about something.

View on HN

I find the 5 star system on Amazon useful. I don't always look at the average rating, but rather the distribution graph of ratings.

For example, one common shape looks like this:

   5 stars: |||||||||||
   4 stars: ||||||||
   3 stars: ||||
   2 stars: ||
   1 star:  |||||||

There's way more 1 star than 2 or 3. That usually indicates the product shows up DOA for a lot of people, or is otherwise probabilistically totally unsuitable for some group of people.

If the number of star ratings follows a more normal curve like the following, then I'm more confident in the quality of the product, even if both have an average 4 star rating:

   5 stars: |||||||
   4 stars: ||||||||
   3 stars: ||||
   2 stars: ||
   1 star:  |

This would make a fascinating study of reliability - can you determine statistically if a product will be reliable based on rating distribution.

You could study Amazon, App Store ratings, etc. Cool insight!

I wouldn't take any rating metric seriously unless it accounted for how long they've owned it. Plus there's the difference between "I've owned 12 lawnmowers and this one's pretty okay, only broke down after 1.5yrs, 3 stars" and "this is my first lawnmower, used it once, it's amazing, never going back to cutting my lawn with scissors, 5 stars". Plus niche sites sometimes have better reviews for the same products (newegg).

All that said, there is information to be harvested in some of these reviews.

Good point about niche sites and the length of ownership. I've noticed that a number of starts on products like top rated power supplies goes down after a year an a half or so. While you might be getting a newly released power supply with 500 reviews averaging closer to 5 stars, after 2 years, you will notice that same product with 1600 reviews averaging at below 4 stars, because owners probably came back and updated their reviews.

It would be interesting to see a trend line of change in sentiment about a products over time, using existing data.

It's been done, and I've seen it as a worked example in introduction to Bayesian stats pages, maybe some of which were posted on HN.

The other common pattern being shill reviews:

   5 stars: ||||||||||||
   4 stars: ||
   3 stars: ||||
   2 stars: |||||
   1 star:  ||||||||

> most people think in absolutes: Either something is the bee's knees or it's fried shit on a biscuit. (Yes, this is from two levels up.)

There was a fascinating post to the OKCupid blog discussing the distribution of attractiveness ratings (also on a 1-5 scale).

Ratings of women by men looked like this:

    5: ||||
    4: ||||||||
    3: ||||||||||
    2: ||||||||
    1: ||||

Ratings of men by women looked like this:

    5: ||
    4: ||
    3: |||
    2: |||||||
    1: ||||||||||||||||||||

EDIT: As you might expect when working from memory, I got some details wrong. Post is here: http://blog.okcupid.com/index.php/your-looks-and-online-dati...

Obvious corrections:

Scale is 0-5, not 1-5

Curve for women is not as steep as depicted in my comment. However, the modal rating is 1, followed by 0. In the words of the post, "women rate an incredible 80% of guys as worse-looking than medium. Very harsh."

A lot of times, the 1-star ratings are only "knee-jerk reactions". The 5-star ratings are sometimes added by people related to the product (or people who for some reason have an agenda). I usually drop those two and focus on the 2-4 reviews.

Who rates something 3? Why bother to interact with a UI to express utter meh?

While you both have a point, I'm not sure it makes a difference in practice.

OK, granted, I've been an Amazon.com customer since 1997 or so, but to me a 3.5 star rating is like writing BEWARE! in flaming letters across the sky. I never buy something that poorly rated before thoroughly investigating what's behind it. E.g. Kindle version's content is messed up I discount since I don't play in that ecosystem except for freebies.

While there are plenty of mostly bi-modal distributions (5 or 1 stars), for lots of things there are carefully chosen ratings in-between, often telling you why it lost a star or 2. Even ones edited some time after with additional commentary and often an adjusted score.

So I'd say it gets the specific job done, which is quite different than YouTube's where the cost of a wrong choice is very very small. I find the reviews invaluable, to the point where there's a lot of stuff I won't buy unless it has enough quality reviews for me to make an informed judgement.

http://xkcd.com/1098/

Netflix does a great job by showing hint text of "Hated it" for 1 star. It certainly helps calibrate people, rather than having individuals come up with their own meaning of what 1-5 means.

I wonder if a five star rating system could secretly count only the 2 and 4 star reviews without telling anyone, sort of filter out anyone passionate enough to try to game things...

I think the human mind is best at that. Even if there are too many reviews for you to completely read, you can sample them, especially the ones others have rated for usefulness, and with some practice recognize the ones that are passionate with sufficient support.

Specificity is one think I really look for. Those who say essentially nothing more than "It's wonderful" or "It sucks" are noise. Those that say, "It's great, but I'm subtracting one star because the control console is a pain because of X, W and Y" are golden ... especially when people add comments on workarounds....

It's kind of funny, in that this "impersonal" mail order outfit very often manages to provide a better shopping experience than brick and mortar stores for those of us willing to buy things sight unseen. Heck, it's not uncommon for me to check Amazon.com reviews before buying something from one of those.

That might work, but I have doubts. Interestingly, it's the opposite approach the the one taken by pre-employment personality tests, in which each question has no partial credit and exactly one right answer, which is either "strongly agree" or "strongly disagree".

Oh right!

I think Barbara Ehrenreich mentioned that in Nickel and Dimed. At Walmart, one of the hiring surveys included some question pitting personal considerations or ethics against loyalty to management, and the only correct answer was absolute loyalty to management. When she debated the point, she was not offered a job.

In college, I got a part-time job for a stretch of some months grading exercises at the local Kumon center. The job consisted primarily of comparing a filled-out exercise booklet to a printed list of answers and marking incorrect answers. (The other part, the good part, was talking to the kids and trying to help them with problems they didn't get.)

My boss was never happy with the speed at which I worked. Eventually she told me I was too smart for the job and should aim higher. In a vaguely similar way, I don't think Wal-mart is looking for their lowest-level employees to make a lot of independent ethical decisions on the job. If you're going to be hiring unskilled labor, you might not want them to exercise a lot of discretion, on the theory that they're at least as likely as not to screw it up.

IMDB (which is a 10-star system) has a 'secret algorithm' that fiddles with the raw data to come up with a computed score. It's been suggested that some of the factors involved include reduced weight for 1 and 10 star ratings (as well as the age of the user's account - votes from new accounts don't get as much weight, etc).

I'd guess that moderation (as measured by 2-4 as opposed to 1 or 5 stars) is a worse predictor of an opinion's relevance than even the most simplistic automated classification of textual or behavioral feedback.

up/down systens are stupid since they are so binary. It only remotely works to give you an actual picture of what people think about it. Having a five scale score is very logical, since it's symetric (and therefore not biased like the 6 scores rating you suggest if you could rate without stars), and you give a degree of satisfaction and not just up or down. It's way superior to tell you if the product is exceptional or just "meh".

Of course you have to look at the ratings distribution and not just the average. The average tells you almost nothing.

Reply on news.ycombinator.com