It's amazing to go read a novel rated 3 stars (because it operates outside this loop) and find it transcendently good.
It's how the "official" reviews from Marriott/Hilton/etc all work
2. Ask each person a different objective question, intentionally filling in the gaps.
3. After a few people, you have an objective, relatively thorough view of the house, but you've not burdened any of your users with more than one question per stay.
The problem with star based rating systems is that they're INCREDIBLY subjective. I oftentimes read yelp reviews that are 3 star reviews where the person raves about the restaurant and then complains about a minor issue. I have a similar issue with app store ratings where I've had a rash of people rate one of my apps 1 star with the review text reading "I just downloaded this, I'll raise my review to 5 stars if love this app".
It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback.
Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?
I know I personally gave plenty of three star reviews on Lyft before I learned better, meaning them as "no complaints". Who knows what happened to those drivers as a result. Why even invite the misunderstanding?
In France for instance, it's very rare for a student to go above 15/20 or so at the college level (I graduated in the top 10% of my class with a 12/20 average; the top ranked student, whom I was friends with, had a 14/20 average).
But when I graded homework as a graduate student for an american university, the professor told me I was too harsh, and after adjusting my criteria to match what she wanted, I found myself giving many 100/100.
When it comes to ratings used by others to evaluate, no system is more customer and vendor-hostile than one where everything that meets a minimum threshold for adequacy should be rated as perfect.
I guess, at least a small step would be to make it so values less than the highest rating don't count against you so negatively - it should be ok to be a 3/5 or 7/10.
1. normalize reviews to make a 4 star from someone who always gives 5 stars worth less than a 3 star from someone who always gives 2 stars; and
2. make reviews secret?
Perception of books and concepts covered within are entirely subjective.
However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.
No, all assessments of value are subjective (and for most services involve subjective weighting of multiple different subjective quality factors.) And the mapping from an internal value judgement to a star rating is something which is a separate subjective evaluation, and one on which research has shown substantial cultural variation along ethnic and other lines.
The rating may be inaccurate because you can only measure in full stars, but it'll still be objective and the inaccuracy is the fault of the rating system, not me.
Objective (adjective):
> (of a person or their judgment) not influenced by personal feelings or opinions in considering and representing facts.
Hmm. I'm willing to accept that the average reader is a more sophisticated and thoughtful consumer than for some other product categories. And there may be less overt tribalism and fanboyism in book reviews.
But, in my experience, there's still a lot of people who will almost unthinkingly give 5 stars to a favorite author or genre. (And I know some of those people who are pretty uncritical about SF books or whatever.)
Again, novels and products are entirely different in that you're hopefully shopping for a product by its specs. You can't 'research' novels beforehand (unless you count as reading other novels by the same author as research) and they don't have any specs to go by.
On the popular restaurant review site Tabelog, a 3.6 (out of 5) is a very good ranking; there's more 3-star Michelin restaurants in Tokyo than restaurants above 4.5.
If you do it secretly like shadowbanning and introduce it softly by slowly blending between conventional score and normalized, it would drive a whole generation of SEO/social/share-economy experts delightfully mad. Think of all the free publicity!
Jokes aside, it might help to regard low ratings as what they are: a way to flag bad behavior, dead serious stuff. We should stop pretending that it is a game, then maybe solutions will be more likely to appear.
There are particular instances (like in this case) where there are a lot of incentives for just about everyone to give 5 stars, 10 stars, or a positive and call it a day.
But, in general, there are issues with self-selection, bias toward the high end of the scale, ratings meaning different things for different types and classes of product, etc.