Hacker News Reader

Comment by LocalH | Hacker News Reader

My SO is a restaurant manager, and she has told me that the only reviews/ratings that have any positive effects are 5s out of 5. 4 out of 5? You might as well have put 0 or 1.

It's extremely toxic.

View on HN

I remember once I bought a car and the salesman said if I had any reservations about giving him a 10/10 on every question in the survey no matter what it was that I should text or call his personal phone and he'd do whatever he could to fix it, also directly saying that it's better for them to have 95% of people not fill it out at all and get 10/10 from everyone else than to have 100% completion with a single 9/10.

I don't know if it's true or not but he told me he lost out on a $10k volume bonus because someone who was extremely happy with everything gave him 9/10 on all questions and in the comments simply put "I wouldn't change anything about my experience, it was great! But perfection is reserved for God."

I've heard this from everyone I've ever bought a car from, everyone I know who has ever been involved with that industry, etc.

Why even have surveys if the point is just to collect 100%s and not actually find things to change without taking money out of people's pockets?

This is classic Goodhart's Law stuff[1]. The key takeaway is that any company that implement these kinds of incentives doesn't actually give a shit about customer experience, they just like to play pretend.

[1] https://en.wikipedia.org/wiki/Goodhart%27s_law

From the story it seems like they just like to deny bonuses on flimsy pretexts.

No, the key takeaway is that they probably shouldn't be relying on ratings to the extent that they are.

When I'm buying a product on a site like amazon with 5 start rating scale, I find that all of the useful information is found in the 2, 3 and 4 star reviews.

1 star reviews are mostly people pissed off for some reason, people with unreasonable expectations, or competitors trying to hurt their competition. 5 stars reviews are either zero effort reviews from people with low expectations or they are outright fake reviews from people incentivized to inflate the rating.

The 3 star reviews identify key and serious problems with the product that the customer otherwise would have liked. Usually the most information is found here. 4 star reviews identify weaknesses in the product that the customer still liked despite the drawbacks. 2 stars often are the same as 1 star but from less critical people.

The only signal I pay attention to from 1 star and 5 star reviews is if there are too many of them it's a red flag. More than 10% 1 star usually indicates a serious problem with product defects. More than 80% 5 star indicates a product that's buying fake reviews.

Or that is just a simple theory that is easy for you to understand. 80% 5 stars might mean something else and 1 stars might mean as much as 2/5.

Of course it's a simple theory that's exactly the point. It has explaining power though and I find it useful YMMV. What I was trying to get across is that you learn more from the text of the reviewers who voted somewhere in the middle. The 5 star reviews rarely bother to say more than a sentence and they rarely include any useful criticism. The people giving 1 stars are often just pissed off for some reason that will not affect me. Stuff like "It wasn't what I ordered" or "they sent me a used product" - The signal is in the remaining reviews if you take the time to read them.

> Why even have surveys if the point is just to collect 100%s and not actually find things to change without taking money out of people's pockets?

"So if I implement this system then either I get confirmation that everything I'm responsible for is perfect or I get an excuse to save money on labor?" - management approximately 8 seconds before implementing the surveys.

This mirrors my experience in retail. Anything less than the top score is counted as a negative, often counting the 1-9 score as a 0 with the 10 as a 1. The heat lands on a local manager when corporate comes to town. What ends up happening is that the customer-facing employees train the customers to put down 10s.

It's ultimately a metric that's been gamified by everyone involved. I think the cynics keep it around because it gives customers a voice.

Which is ironic, given that it modifies the customer's actual voice when interpreting the numbers in such a toxic way. I point to the cousin comment where a salesman lost a $10k bonus because a customer didn't give them a 10, despite the customer's comment being very explicit that they were extremely happy with the experience, but that they refused to rate anything perfectly, as they reserved that for God. Despite that comment, the 9 was used as justification to deny the bonus, effectively also denying the customer's voice.

Or pull a YouTube/google and just eventually totally remove the option to down vote at all. “ Yeah… we’ve heard the vox populi and ehhh, gonna have to pass on that one.”

And Ebay, so 20+ years. It was either positive and A+++++ or it's effectively negative.

To me that’s the crazy thing. You give a 4/5 and immediately you’re asked what was wrong.

It’s also why Netflix moved tot the thumbs up/down system. When they moved into new territories their recommendations system had troubles since some territories would rate everything as “bad”, because in that territory 3/5 meant it was fine and they enjoyed the show.

Reply on news.ycombinator.com