The Problem with Feedback
theatlantic.com
theatlantic.com
It's more likely they'd lose their job if they consistently receive bad ratings, from several different customers. And they should.
Given how much it costs to bring on new employees it's more likely that one bad review results in a consultation, additional training, etc.
Customer satisfaction surveys can be a useful tool, but they're often applied as a blunt instrument on employees who have no control over the situation the customer is dissatisfied with. It's basically a lazy copout as opposed to actually fixing company policy. I tend to ignore them unless it's to give good feedback.
A more humane system would take the feedback and then interview the employee in question and their management to understand what the root problem is. But the management organizations that use these surveys are almost never humane. Plainly, they don't give a shit about any of their employees.
I picture myself being let go from my job, I begin to feel the stress of doing the first inkling of planning that would be needed to survive that, and I decide that I didn't have such a bad time that I feel like pressing that one-star button, so I just close the feedback page instead.
I wouldn't want people to rate me one-star, so I don't rate other people one-star.
Until we have UBI, or a functioning safety net that universally guarantees shelter and food, this is false.
Assuming I care how they respond, there are a few benefits: - people are conditioned to rate at the extremities with a 10-point scale, whereas they tend to use the whole spectrum more when the scale is reduced to seven points. - most people don’t realize that “1-10” has no true middle number. On the other hand, 4/7 is perfectly “in the middle”. - if you believe psychologists, the average persons working memory is “7, +/-2” items. In my opinion, the brain seems to treat the “2” and “3” responses on a 10-point scale as roughly indistinguishable. There is similar haziness between “6” and “7”. The average person seems more capable of making distinctions across seven points.
Anyway, when you’re asking for feedback (and not on a first date), consider the seven-point scale!
I don’t frequently ask friends to rate their interactions with me. ;)
Perhaps I should.
Uber doesn't want your feedback because they care about your ideas/opinions. They want it because it's critical to their business.
Put yourself in their shoes. They have 75 million users and 3 million drivers. A critical part of growth is retention: not bleeding the existing user base you have, by offering a quality service.
So how do they identify undesirable drivers to maintain the safety and quality of their service? And in a scalable way...
You ask users to give drivers a review out of 5 stars. Chances are, if multiple people give a driver 1 star, others will too. It's a pretty good proxy for driver quality.
I can definitely see how this feels "dehumanizing" and like you're a "cog in the system". You are a cog in the system. You're a cog in their automated system for keeping their service safe and high quality.
There are different kinds of user feedback, for different reasons. One kind is user interviews, where you want to hear people's ideas and opinions for how you can do better. This is pretty human. Reviews are pretty automated, and just a different kind of "feedback".
If you run a consumer company (as opposed to B2B), it isn't practical to conduct user interviews on even 1% of your entire user base. So it isn't surprising that giving feedback to consumer companies tends to be a pretty negative experience.
[1] https://canny.io
- Driver calls and asks where I'm going then cancels ride
- Driver drives up to me and asks where I'm going then cancels ride
- Driver drives around aimlessly for 40min trying to get me to cancel and then cancels ride
- Driver for Uber eats picks up food, steals it, then cancels ride
In all of these instances the cancellation removes the ability to rate - these are the cases when I'd want to rate zero stars.
If the person picks me up and takes me where I need to go I'll rate them five stars, in the worst case I just won't rate them.
I live pretty close to the airport (6-20 mins) so the sneaky drivers that see the fare will call me and confirm my destination. I confirm and then they just sit in their waiting spot indefinitely, hoping I cancel.
Understanding their strategy, I walk down to the lower level with the app still open and take a cab home. They see me physically moving (haha) and have choice but to cancel. I leave the app open even when I get home.
I even had one driver listed as hearing impaired who called me up to confirm my destination. He had no trouble hearing and confirmed he would be right there to get me (only to just wait it out so I would hopefully cancel!)
Next time - leave the app open and use another service or grab a taxi. They'll see you move on the map and you'll waste their time since they were trying to waste yours. :)
Also - I always tip about double the norm for any driver that picks me up and doesn't have a bad attitude because I know they would have preferred a longer trip. I shouldn't have to do this, but I'm a business owner and I don't mind rewarding those that do the right things.
In those instances I've put my phone down and waited them out, but Uber still charged me (even though they're not supposed to). I was able to get them to reverse the $5 fee after some arguing, but was still unable to negatively rate the driver.
Depending on how much they wanted to get more uniformly distributed ratings, they could (a) make the rider rating depend on the uniformity of the ratings given or (b) just use a comparison rating ("Was this better or worse than your previous trip?").
I'm also very interested in the topic, but in a slightly different domain. I'm on a mission to get "was this page helpful?" feedback widgets on all technical documentation across the web. Recently I just helped the Kubernetes folks [1] add it to their docs site.
One of my high-level goals is to get the docs community to aggregate all of this data in order to benchmark typical ratings for common types of documents. E.g. what's the typical rating for a tutorial? I also want us to start actively experimenting in order to validate what kinds of changes lead to meaningful improvements in documentation. E.g. adding examples to conceptual overviews improves the rating by 5% (not real numbers).
Find me on the interwebz if you'd like help implementing one of these systems in your docs, or want to join this project.
> "If thumbs-ups or ratings on a five-point scale are not automatically useful, what kind of feedback would be? Finely tuned feedback that targets the system it’s meant to regulate will always surpass a barrage of angry or ecstatic reviews. Rather than trumpeting the desirability of all feedback, apps and review sites should pursue only the information that is crucial for making the system work better."
It's hard to know what data points correlate to making the system work better until after the fact — so developers generally have to "over ask" so you can back test data to outcomes.
Probably the best criticism of the proliferation around feedback is that it doesn't always make the customer's experience better, it just creates better conditions for the company to make more money.
This is especially true if you have retention numbers, already. Why bother asking if I was happy with a purchase, if I never buy again anyway.
Note that I am sympathetic. There is every chance that I will buy something again simply because the exit survey reminded me of your existence. There is also the chance having an exit survey will make me not want to come back. :)
Now, can you mitigate some by carefully doing an experiment again? Certainly. But you should also not constrain experiments to only things to have been measuring already.
Does it really if it does? Or is people __believing__ it does good enough? If your comfortable with the latter then such feedback will create confirmation bias. Which the product dev team (also being human) is also going to fall victim to.
Traditionally, this interpretation has been done wholly by humans, and was a long and laborious task. However, we can now do much of the manual work with NLP [1] - discovering topics of concern and measuring the prevalence and sentiment. This lets us ask the dataset useful questions, like "What issue is angering customers most?", and "What features should we be talking more about?"
It's a move towards a much, much wider sampling of customer experiences (compared to focus groups), and I think it should be celebrated for empowering companies to make products/services that consumers really, really like.
The problem is, these rating lack context. Consider a pop song with a gazillion streams on YouTube. How many of those were "unemployed" 15 year olds pretending to be sick and staying home from school? Without some sense of context those gazillion likes have little meaning.
And of course, baring some drastically awful experience, there's confirmation bias. The truth is most people aren't objective enough to admit they might have made a bad choice. Of course it was the best ____ ever! It was my ____. I bought it.
Even so, there are always idiots who will order pizza at a place that specializes in curry, or curry from a place that does Mexican. Yes, it's on the menu. But that's doesn't mean you should order it and then complain about it.
Just about every time I want to rate the employee as a 5, and the company (or management?) at a low rating because the business has obviously been the one to screw up, not the employee.
Asking to rate an employee is a sign to me that the company service is poor.
I actually might visit a place if I know that a particular group of people strongly dislikes it. People have different tastes and a review system might account for that.
It would require people to fill in a few questions about themselves. Or it needs cross-referenced with Facebook likes or something like that.
This information can belong to the people!
U.S. Patent Pending
And to the author: kybernetes (κυβερνήτης) is the steersman of a boat, not a governor.
I've wondered before if this says something about heavy book readers as a population, since the five star rating system doesn't seem to work that well anywhere else outside of professional critical reviews.
Consider:
1) As a user, won't be satisfied with a feedback system if the company on the other end is evil. Take Uber as an example. The tension between drivers and passengers isn't caused by the 5-star rating itself. It has another critical component - that Uber set the system to squeeze out as much value out of drivers as possible. Since they essentially optimize for bad driver treatment, drivers get defensive, and this screws up user experience. Hitting with 1-stars until morale improves isn't the solution; not exploiting people is. Similar thing happens in some customer service interactions, where rating is - again - not a signal on quality, but on whether or not I hate the other person enough to want them to lose their job.
2) A rating system won't work for you if you connect it to a money-printing machine. This one should have been obvious immediately, and definitely should be by now. This is why Amazon, eBay, Yelp, et al. have problems with reviews. As long as their presence or absence impacts immediate sales, they'll be gamed to the point of uselessness.
3) Some companies like to deploy ratings to skip doing the work, and then they cry foul when the results don't materialize. Again, Uber won't solve driver problems with star ratings; that would require actually meeting and evaluating people. I can hear the screams - "but that doesn't scale!". So what. I want to go to the Moon by lifting myself by my own bootstraps, but I can't, because that's not how physics works. I'm not going to cry that I can't get what I want without doing the work that's necessary to get it.
4) Feedback loops must be actually closed for them to work. I.e. not just collecting the data, but acting on it.
5) If you can't capture the entirety of your vision in directly measurable metrics (and you probably can't), then don't follow them blindly. In particular, be mindful of what you actually measure, 'lest you end up pissing off a lot of users and defending yourself with "data told me so". "You make what you measure" works both ways.
--
TL;DR: Blame the people/companies deploying feedback systems for being clueless or having malicious intent, not the feedback systems themselves for simply working the way they work.