Customer ratings have become meaningless. ‘people give 5 stars like it’s candy.’ | Hacker News Reader

Customer ratings have become meaningless. ‘people give 5 stars like it’s candy.’

wsj.com

44 pointsbalderdash68 comments

It is probably the same mechanism by which people still tip generously even when they are not completely satisfied with the service.

People don't want to be seen as rude and they are afraid of spoiling a probably long-term relationship.

Others maybe are afraid that not giving a 5-star rating will have a disproportionate effect on the seller's reputation, and while they are not 100% satisfied, they don't want to feel guilty about bringing some stranger to potential bankruptcy and potentially making other strangers unemployed.

More tech-savvy and thus more affluent people tend not to feel so strongly about this kind of thing, and at the beginning of online commercial services, they made up the bulk of the customers. Rich people can get away with being hyper-critical in their commercial relations and yet continue to be welcomed by shops. Also, as they don't get so much of this treatment, it is usual for them not to develop this kind of empathy. Someone who has never been laid off or had to wonder how they would pay the rent if they lost their job is not triggered on an emotional level wondering what would happen to other people if the business they work in floundered as a result of bad reviews.

Frequently you are told that anything less than 5 stars will have negative consequences for the vendor whether it is a Ebay seller, Uber driver, making a credit card payment, or the service department at your car dealer. In that case you have no way to call out the truly exceptional from adequate. Adequate does make the world go ‘round.

Frequently I leave no rating because of this and sometimes, if your mobile app is pushing me to leave a rating, I will leave a 1 star rating and say I left that rating because I as trying to complete a task and they distracted me at the wrong time asking for a rating.

The uber app requires me to justify the decision anytime I give lower than 5 stars.

Usually just end up not giving any rating at all.

I gave a Lyft driver 4 stars because they got honked at by a car in their blind spot as the Lyft driver was about to change lanes into them. NBD could have, and probably has, happened to anyone. So 4 stars, "drive safety" or "drive quality" or something and "check your blind spots more carefully next time" as the note. 10 minutes later I get an email from Lyft telling me they'll reprimand the driver and possibly remove him and he's banned from ever matching with me again. So now I just give 5 stars unless something actually noteworthy happens.

> reprimand for 4 stars

Wow, what extremism

Rating inflation isn't caused by customers "giving 5 stars like candy".

It's caused by platforms redefining 4 stars into a failing grade.

And the reason can’t be the Uber platform is broken, or I had 5 drivers cancel in advance

Because business analyst treats anything below 5 as a failure, and often with consequences for the workers. So it's not my job to do HR's, and I don't want to pull the trigger of their gun on their employees.

If I give a waiter/seller/worker.... 3/5 stars, it means that service was OK, a bit above average, nothing to worry about. But the poor guy will be dragged through the mud at his quarterly review, so why would I inflict that unto him?

And finally, selfishly, I have other things to think about outside of where in the scale of every similar workers I interacted with this guys falls. So just take your 5 stars and let me be.

Yup.

I very nearly changed banks over this particular quirk of "Anything below the maximum rating is a failure that must be addressed."

I'm one of those people who likes going into the bank lobby often enough to get cash (though the ATMs have started handing out $100s now so I do that somewhat less), and for a while, I was getting a survey phone call about every single visit. The people at the other end would ask how various aspects of the transaction were, and would not accept anything less than 10 without long explanations of how the bank teller didn't make my day the most amazing ever. They would not accept "I'm entirely happy with the transaction, it was utterly non-exceptional, the money was counted properly, and this is what I expect" as a reason for not deploying fireworks of celebration in my rating of bank encounters.

I mentioned this to the branch manager at one point when I was getting a wire transfer set up (at the time, at least, wire transfers to other countries had to go through a few hoops to make sure you weren't getting scammed), and that I was very seriously considering switching banks over this annoyance, and he very rapidly put me on a "Do not pester, no, seriously, don't..." list that solved the problem.

I just generally don't rate stuff anymore, because I don't consider "5 stars" a "meets expectations" grade rating.

It's funny that educational institutions often get chided for "grade inflation", when grade inflation is the norm in industry.

This is one of my biggest pet peeves, and you see it in so many different places whether it’s restaurants, wine, hotels, etc.

basically, all ratings do for me now are truncate outcomes, stuff that’s rated really badly is probably best to be avoided, and stuff that’s rated really highly is unlikely to be horrible. but that’s about it, more often than not, I’d classify things rated quite highly as average/mediocre, and far far short of “excellent”

Yes, personally I find that most (with a very few exceptions) transactions are "honest", I expect to pay x for x services, or y for y services, if this happens there is nothing to be said on the matter.

In an appropriate scale these, let's say 95% of total experiences, would get a 3 or 4 stars rating, the 5 stars would be for the - still say - 2 or 3 percent actually exceptional, and the 1 star for those 2 or 3 percent that are really a total disaster.

Japan has a reasonable standard for reviews. 3 stars means the place met 100% of expectations, it was totally average. Nothing bad but nothing extraordinary, a reasonable place to spend your money that you will enjoy. 2 or 4 stars is like 90% or 110% expectations met, and 1 or 5 stars is like 50% or 150% expectations met. It’s much easier to scan a map for new places, and Japanese standards are already a bit higher than Western ones in terms of orderliness and service. I’ve been to restaurants with 3 stars (an abysmal rating in America indicative of impending doom) that were clean, delicious and friendly.

This is how it should be! That would turn ratings into something roughly useful.

As it is, I even had an argument with an Uber driver here, where he was saying that anything less than a 5 star rating is a bad rating. To me, a 3 star rating is "acceptable, not extraordinary". You don't get into "bad" territory until you're lower than 3.

I just can't go there, because if the Uber driver is correct, then what's the point of ratings at all?

The problem is that we have decided above average is not longer sufficient. Everything has to be exceptional, and if it is not, someone needs to get fired.

I have a similar anecdote from my Amazon days. Amazon started a daily survey for all employees (known as connections). It started out as a survey and very soon morphed into a tool to evaluate managers. Managers would often get comments about how their team is rating the work culture poorly and it is their responsibility to improve things (even if it is beyond their control). Everyone was/is gaming the system.

My advice to my teams was to treat it like a Uber rating - * Everything is going well, it is a 5. * Something minor is bad, but you can talk to your manager to close it, it is a 4. * You are moving out of the team due to your manager, it is a 3. * The entire team hates the manager, it is a 2. * The entire team hates the manager and everyone else hates the manager, it is a 1

Interesting strategy because it implies coordination for 2 and 1. How do you know everyone hates the manager for example?

Engineers talk to each other and they are smart. As long as enough people perceive the same thing, it works out.

> Everything has to be exceptional, and if it is not, someone needs to get fired.

Every company with 100+ employees I worked for had a 'stack rating', so yes, people were/are getting fired constantly. Perhaps not a good analogy, perhaps it is, but yes, I genuinely think 'someone needs to get fired'.

One reason people give 5 star ratings is because they are asked to explain why if they rate lower. If companies made '3' the normal rating and asked for an explanation for higher OR lower ratings, that might normalize the ratings better.

Exactly my thoughts. It works well IMO and experience. Our company has this same policy for yearly feedback ratings.

Most people know that if a car share driver gets enough bad reviews they get kicked off the platform. Maybe part of that trend is not wanting to be responsible for "taking away" someone's source of income, even if the service was just average. Really egregious things can be reported in other ways.

There's also the cognitive burden of having to compare "this" ride to "every other ride you've ever had". That, plus the fact that you rarely see an impact of giving reviews... it's easier to just hit 5 stars or ignore.

As someone who used to work in a technical support call center, I understand just how much control these metrics can have over someone. Anything less than a 10.0/10.0 CSAT was considered not good enough at times.

So today if I call and I get an agent at a call center that is even moderately competent I give them a full score to help them out.

This is what I do. Uber drivers, call center agents, any place that has obvious metric driven job performance evaluations gets a 5/5 from me unless the interaction was truly terrible.

On top of that most front-line workers have little control over circumstances that affect me. If my internet is down because of maintenance, it's hardly the fault of some dude half way across the world. Yeah, it sucks, thanks for giving me an ETA of restored service. 5/5.

I've given 5 stars to Uber delivery drivers who are 40 minutes late. I do this because I know that anything that's under 5 stars will have such grave consequences for this person that it's just mean. It has grave consequences because of people who think exactly like me - eventually the standard for what constitutes 4 star service is so incredibly low that rating someone 4 stars is practically telling them to go fuck themselves.

When I moved to America years ago I kept getting followup phone calls/emails "we see you gave 3stars and wanted to check in with you".

It's 3 stars. Above average. Why are you following up?!

4 would be exceptional.

5 is reserved for absolute perfection.

Why are you bothering me with this phone call.

I then learnt Uber drivers don't get rides unless they are above a 4.7. sigh cultural differences and all. May as well move everything to a thumbs up or down if your going to use the rating that way.

Bingo.

Companies actually reach out about 3 and 4 star reviews, almost like its an insult... I think users have been trained to feel "guilty" about sub 5 star reviews.

Not these ones:

https://uk.trustpilot.com/review/www.myringgo.com

When companies have reached out to me like this and asked me to change my rating, I have always accommodated them by dropping a star off what I initially gave.

Reach out to me to ask how my experience was? Sure, no problem -- as long as you leave me alone if I don't reply. Reach out to me to complain about my rating? That's just harassment.

Phone calls and e-mails are horrible, but where did you get the idea that 3 stars is above average?

On a scale of 1 to 5 stars, 3 is in the middle. In a normal distribution, that is average. 4 stars would be above average, 2 stars is below average.

Although many people would even view that as overly negative. These days, most people interpret 5 stars as basically "met all expectations, zero complaints". Then 4 stars means there was something wrong but still ultimately did the job fine, 3 stars means serious problems but still had some usefulness, 2 stars is nearly useless, and 1 star means totally useless.

Edit: And I don't understand what "absolute perfection" means. If a delivery driver shows up in the expected time window and rings my bell and hands me my package, that's 5 stars. I'd give probably 90% of deliveries 5 stars, for example -- there's literally nothing for the driver to improve upon. Do you consider that "absolute perfection"? (If they don't ring and they leave it on the stoop when I'm home, that's not 5 stars.)

Well at or above average for 3 stars.

The view that it's overly negative is a curious cultural norm.

> where did you get the idea that 3 stars is above average?

Because people do not think of stars as intervals, but as their upper bound reached. In this case, the average would be 2.5 – whereas yes, if you consider stars an interval, then yes, the 2-3 interval is average, but virtually no-one thinks this way.

When you're giving stars there's no such thing as intervals or bounds. You have 5 discrete options from 1 to 5.

Perhaps the confusion here is on whether it's possible to give 0 stars. If you're rating from 0 to 5 then you could consider the midpoint to be 2.5 rather than 3. But generally things like Uber and Amazon have a minimum of 1 star, so the average is not 2.5, but 3.

Yes, but stars are not punctual on your screen. What people see are the stars filled up to the third star, whereas the average score would look like the third star being only half-filled.

No it doesn't. The lowest score looks like one star filled, the highest is 5 stars filled, and the midpoint is 3 stars filled.

If the third star is only half-filled then that would be produced by a mix of half 2's and half 3's. Not all 3's. All 3's results in three stars filled.

>5 is reserved for absolute perfection.

To me, 5 out of 5 has always meant "in the top 5th". It makes sense to me that each number covers the same amount of ground on the scale. I get the psychology behind reserving the highest number for only the best of the best, but I can't think like that.

Regardless, I agree that it's ridiculous that we have a system where anything less than 5 out of 5 is somehow an indication of a problem so serious that the recipient feels justified in contacting me about it. That's an insane system.

One problem is that people just give five stars by default. "I didn't find any insects or hair in my food. Five Stars." Another problem is that there is no consistency. People who only ever eat at McDonalds give all the McDonalds near them four or five stars and people who only eat at Michelin Star restaurants give all of those four or five stars. There's no standard across restaurants. So my solution is, I'm going to actually visit and review every restaurant in my town (there are about 30 of them) and I'm going to use a consistent scale. 5 means wow, this is genuinely the best I've ever had in this genre. 4 means very good, above average. 3 means decent, average, worth your time. 2 means not great, but edible. 1 means don't go here.

I think that the people who rate McDonald's 5-stars are probably being more helpful about the system than you are, because "restaurants" is far too broad a category to have a consistent scale within.

Instead, it's more useful to rate a given restaurant compared to its sub-genre (or at least its price band) -- was it good for a fast-food restaurant? -- so you might rate a disappointing experience at a Michelin star restaurant lower than you'd rate a clean McDonald's with fast service, just because of how they're meeting your expectations.

This results in usable review aggregates. If I look at a Yelp! listing for a McDonald's and I see it's 2-stars, I want to know that's because it's bad for a McDonald's, not just because the person rating it thinks that a McDonald's inherently has a cap of being "not great, but edible".

In addition to not rating within a category/genre, I actually find that people tend to overweight meal price in their weighting (e.g. an expensive but average Italian dinner gets a higher rating than a phenomenal and cheap slice of pizza)

We need to go the route some professional reviewers and online platforms (like Steam, for instance) have taken: no stars.

Recommend or dont recommend. Write some specific feedback, or don't write anything at all.

It's a shame that ratings provide so much leverage over employees.

A business I worked for sold their software items on a platform that used to do rating removal when the review didn't provide any context but eventually either bad rep for doing that caught up to them or the scale grew so much that they could no longer afford going through each case and they changed their policies. People would give bad ratings for the worst reasons:

1. They need a refund but the reason is along the lines of "I thought I was buying a different item". They can't ask their bank to do that because then they'll be banned on the platform. If the business provides a refund, then it takes a loss by refunding the part of the price that goes to the platform.

2. They haven't been able to figure out some very basic functionality and they can't even tell you what exactly went wrong. When you try to help them, they either never respond or you help them figure it out and then they don't change their rating.

3. They demand you customize the item to have functionality that it does not have or show it its marketing, you turn them down and suddenly your item is the worst on the market.

Of course there were constructive bad ratings and I'm not saying that the business was selling the cure for cancer, just providing my perspective. It's a tough world out there :)

It's not meaningless. You just have have to reset the scale based on the platform.

On Amazon, for instance, you have to ignore the first 4 stars. So it's:

Under 4 stars: total garbage

4.1 = 1 : a little better garbage

4.5 = 5 : About average

4.9 = 9 : Amazing

My wife and I have a rule: Books have to have at least 4.6 on Amazon and 4.2 on GoodReads (which is a harsher platform).

The Amazon scale above (ignore the first 4 stars) also works for Uber Drivers. An Uber with a 4.1 score is a danger to life and limb.

I like the goodreads rating system because they give you hints when you are reviewing a book and hover over the ratings.

I believe 3 stars is "I liked it", 4 is "very good" and 5 is "exceptional" or something similar.

Which is exactly what those ratings should mean for everything.

Worst case of this is online reviews at Home Depot. Every review is five stars even when the text is something like "it spontaneously combusted as soon as we brought it in the house and killed one of my children."

Thanks, Sir. You made my day.

And it is even funnier exactly because it is very close to the truth.

Reviewers have learned if they say anything remotely negative AND give less than perfect stars, the review is found “wanting” or “in violation of our code of conduct” and never published. So people work out how to game it.

Sounds dumb maybe, but what if there was a system to give a 6th star with the disincentive of requiring the user to give a very detailed review (including photos and whatnot) tied to their verified order and payment info why they are awarding it?

This wouldn't just disambiguate the adequate from the truly great, but put the seller in direct contact with their best customers.

In some ways this already exists in the delivery apps, but needs a lot more polishing up.

Not a dumb idea, but I'm sure it would just lead to further rating inflation. If there was a 6th star, guaranteed all the company's "measurements" would quickly morph to consider only 6/6 as acceptable and everything from 1/6 to 5/6 would be considered bad.

Yeah they can try, but that's the point though.

The reviews are required to be detailed instead of just stars and requires real orders from real users so it increases the signal to noise ratio.

Also, legitimate businesses would be far more interested in the sales leads from these 6-star reviewers than impressing the average consumer with spam. There's aligned incentive for everyone to really care about that 6th star and put it under the microscope when they see one.

The delivery apps have the "top rated in your neighborhood" and "most beloved" tags that already more or less work like this. I'm just saying to generalize it across the web. Maybe there's just something about food that makes people more likely to review it carefully, but I've always found them to be more reliable than the junk on amazon.

I was on the road yesterday and stopped at a McDonald's that offers you a free value meal if you submit a survey and give them top ratings.

Considering it took 28 minutes to get through the drive-thru, my drinks were watered down from melted ice, and they were missing two items, the gaming of corporate survey mandates make the results meaningless.

I would argue that aggregate rating have always been (approximately) meaningless, and it's silly to assume they would ever carry a lot of signal. This is just an instance of Goodheart's Law[1] at work. Most of the financial incentives in these systems are aligned against sincere ratings - and this even applies to the companies building the rating systems. Uber doesn't want to kick off bad drivers, it reduces the supply of labor they can tap into. Airbnb doesn't want low-rated hosts, it reduces their revenue if fewer people stay there. There are long-term consequences, of course, traditional Taxis and Hotels never went away and still pose a competitive threat. But secondary effects like loss of trust in a platform take a long time to play out.

[1] "When a measure becomes a target, it ceases to be a good measure"

https://archive.is/eH4qD

It is not "the people" that are ruining rating systems that were created, run, and controlled by corporate interests.

This is literally someone whining that humans are not acting how they want for more profit from their flashing lights money making machines.

Maybe we need a ranking system based on asking "Compared to your previous transaction, which was a better experience?" Answers "This was better" or "My last experience was better".

Wouldn't the output of that be just a horizontal saw tooth graph?

I'm not sure what you mean by that. Assuming you mostly get a different driver each time, in a fairly random order, drivers who give better experiences would rise to the top. Drivers who give worse experiences would fall to the bottom.

You may have to do some adjustment to account for the fact that most drivers are more likely to be in certain areas, so less likely to be compared with drivers further away. Something like the algorithm they use for ranking chess players could maybe help here.

I mean, let's use a "proper" range 1 to 10.

Your first ride is a 6 (average, sufficient but nothing to talk about), you "upvote" him. Your second one is a 7 (so better than your first one) and you then "upvote" him. Your third one is a 8 (better than your first one) and you then "upvote" him. Your fourth one is a 7 (worse than your last one, but exactly as the second one) but you "downvote" him.

Drivers #2 and #4 were at the same level, but one got an "upvote" and one got a "downvote".

From me one got an "upvote" and one got a "downvote", but from all customers on average they would get a similar number of "upvotes" and "downvotes".

(You can't "upvote" your first driver as you have no comparison)

The best way to counter this is by asking a few questions, and seeing how they rate things across the board.

If you see a perfect 5 stars for Location, Convenience, Value, Cleanliness, and Customer Service, you’ll know that the experience was pretty average and the person is just being nice.

1 star for everything, somebody is just cranky. You should still find out why, but that doesn’t mean that your customer service rep should have their job on the line.

3-4 stars for everything, but customer service gets 1 star or 5 stars? You know you either got a problem, or a rockstar, on your hands.

Customer rankings are, in my opinion, completely without value and should be ignored. Not just because people tend to be binary about them, but also because it's impossible to know how to interpret them.

Is this product 4/5 stars because it's actually great, or is it because it's really cheap, or because the seller shipped it quickly, etc.? There's no way of knowing.

Add in all of the faked, bribed, or purchased ratings, and there's nothing left in the system that is of any use.

Few things are quite as annoying as seeing a 1-star Amazon review for a product that says "this was great, but it was delayed for delivery so I'm only giving it 1 star".

There’s weird begging around customer service ratings too, just like there is for tipping, in my experience - you’ve got people unironically saying “if you don’t give five stars every time you are literally killing the wage slaves who’s survival depends on you lying to their employer” and “if you don’t tip 20% every time -“ in the same breath.

Completely ruins any semblance of legitimacy in those systems.

Officer efficiency ratings in the US Military have been that way for a long time--there are WW II novels (_The Caine Mutiny_) that refer to that.

I just give 5 stars cause I think it's funny to undermine the rating systems.

My mom worked customer service for an insurance company. She was told that her Q2 (Question 2, on a scale of one to ten how satisfied were you with your call?) rating dipped below a 9 it was grounds for termination.

I know some delivery workers and if they get anything less than "excellent" they get written up. So it's "excellent" each and every time

I know it's meaningless but it takes one click to help someone

https://web.archive.org/web/20230605140423/https://www.wsj.c...

You should always be given an opportunity to rate the survey.

Relevant xkcd: https://xkcd.com/1098/

Reply on news.ycombinator.com