Asking the crowd: does anyone know of an in-depth comparison of rating systems (thumbs, 3 stars, 5 stars, 1-10, etc)?
Also obligatory Black Mirror reference: Nosedive S3E1
Also obligatory Black Mirror reference: Nosedive S3E1
1 - Eek! Methinks not.
2 - Meh. I've experienced better.
3 - A-OK.
4 - Yay! I'm a fan.
5 - Woohoo! As good as it gets!
But the subjects of review seem to interpret the ratings like this:
1 - A black hole.
2 - Don't bother.
3 - If you're desperate.
4 - Could be better.
5 - Met all expectations.
To compound the problem, many people actually rate like this, causing what the article calls "reputation inflation." The result is that I end up having to use this higher scale when looking for service, because enough people have been rating that way.
This is why I would prefer a binary up or down vote, with the option to comment. My 1-2 would would translate to No, and 3-5 to Yes.
I've also heard of A. Something was wrong B. Good job—met expectations C. Something was unusually good Sort of like ride-share "give a complement" feature, since they know everyone gives 5 stars.
Another option (that I've never seen) is to tell the rater their response will be weighted based on their previous scores. So if someone who always gives 5 stars gives a 4 star, it's interpreted as a negative response. But if someone who gives majority 3 star reviews gives a 4 it's interpreted as positive.
Basically, you sit down at a restaurant, order a meal, and on the way out, the host or hostess ask you to mark off checkboxes rating the experience on a scall of 0 to 11. Eleven being the Spinal Tap interpretation, and Zero being the hospital visit.
We now face three realities at this point, completely disconnected and untethered from one another, doomed to resolve as if it were some Cellular Automaton rule set from hell.
On the customer's side of things, maybe they let their small child mark the check boxes to satisfy the child's curiosity, maybe it's the waitress' mom dropping by for a visit to take pictures at their first summer job. Maybe it's an actual normal customer, a random person with with a pulse, fresh off the street. Anything goes. Straight A's or all balls.
The wait staff, have no control and don't actually know how they're being judged. By what criteria, or rules. They just know what it means to wait tables. They've been in a restaurant before, and they know what middle of the road is. Unless specifically advised of the level of service, they observe their surroundings, and follow their instinct, drawing on prior experience and whatever they learn as they go.
The analyst that receives the feedback however, operates according to rules both alien and strange to normal people. Only 9, 10 and 11 are positive ratings. All others are an insult to the business, and all parties associated with an 8 rating or less must be eliminated from the system. They don't want to see paying customers that aren't ecstatic, nor do they want employees delivering service that catches a middling rating. To them, a 9 represents a danger zone, threatening profits with a backslide.
The person who explained this principle to me was my boss' boss, and it was in this moment that I knew the ship I was on was aimed at an iceberg, and that I needed to escape. He was referring to our OKR process, while simultaneously explaining this principle in relation to our review process. It was a very "Steve Ballmer/Stack Ranking/Cut the Weakest Link" sort of discussion, and I stuck my thumb out and found another job, for better pay, and less demand within weeks.
Edit: This article has less detail than I remember, but covers the gist (and makes the same Black Mirror reference): https://www.newyorker.com/magazine/2018/02/05/customer-satis...
I guess that makes sense. Individual end-users only ever see the tip of the ice-berg when interacting with a large organization, so they are likely to blame the parts of it that they interact with while ignoring the deeper root-causes. If you let them blame the customer service rep's attitude or the size that their vegetables are chopped then you wind up trying to over-optimize around making the reps smile or definining exactly how to chop the veggies. But if all you know is that people aren't happy with the experience you send a process minded person who is familiar with the whole stack of how things get done to figure out why the service reps are in a foul mood or how come irregularly chopped veggies are winding up on peoples' plates.