Hacker News Reader

Comment by fastball | Hacker News Reader

Eh, Goodreads has always suffered from the same problem that plagues every other review system which uses "score out of X" ranking.

Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage.

Just allowing a boolean rating (ala Rotten Tomatoes when aggregated) is much better, assuming you can get enough reviews for that system to actually work (probably > 30 is required for most applications).

I think "aggregated personal Elo" would be a fun way to rank things: I just give you two books that you've read and you tell me which is better. Do this loads of times and eventually you have a solid ranking of every book you've ever read. Aggregate everyone's rankings and you have a much more robust system then "please rate this book out of 5 stars".

View on HN

Seems like this is culturally different for some things. And corporations can influence too.

I have a friend who has four restaurants in Tokyo, and I've been several times there. If you keep attention to the restaurant reviews in Google Maps, Japanese people is very hard. They'd go like "The food is great, incredible service, surprising flavors, very good experience, best Spanish food I've had in a long time..." and then they go "...but I know they can do better and have space to improve. 2/5".

In Spain we would be like "Nice beer, they gave us some tapas. 5/5"

Gig economy corporations also have made this worse. You're scared to score your driver, your server or your hotel by less than a 5/5 or somebody might get punished or fired (and maybe they didn't even had a contract in the first place).

And this becomes a snake biting its own tail. Now I rarely go to a place with a score of less than a 4/5, and I'll score relatively high because I know I could influence votes out of what people consider "worthy to visit".

> I rarely go to a place with a score of less than a 4/5

I have a simpler heuristic. I ignore all customer ratings entirely, and refuse to leave any.

100% it's cultural. In the Netherlands, companies brag about a 7 or 8 out of ten as proof of their amazing customer service. When I lived in America, it seemed the consensus of anything under a 9 is killing puppies.

Coincidentally, NPS is entirely based on this fact. 8+ is required for a "happy customer". 7 is given a little wiggle room to win them back and under 6 is a lost cause.

> In the Netherlands, companies brag about a 7 or 8 out of ten as proof of their amazing customer service.

Huh, could you give an example of it? I am new to the Netherlands and hadn’t realized this yet.

Sure. I was looking up energy price comparisons last night and came across this.

https://www.overstappen.nl/energie/vergelijken/

Under "Beste energievergelijker"

> Onze klanten beoordelen ons daarom met gemiddeld een 8,6.

If you scroll further, you can see their ratings for other services; none reach even an 8.

Compare that to Nerd Wallet's "Best Life Insurance Companies" (a segment that is nearly identical from all providers if you purchase term life).

https://www.nerdwallet.com/article/insurance/best-life-insur...

5/5 across the board for all parties. I get that it's a "best of" list, but you seriously mean to tell me all of these companies are exactly the same? The rest of Nerd Wallet's top pick lists are the same.

How we rate things in the Netherlands is a reflection of the rating system used in schools.

1-5. Ranging from pathetic to inadequate, in any case you didn't pass.

6. Adequate, the minimum needed to pass. A culturally important number as it expresses the "zesjescultuur", a phenomenon where somebody is intentionally doing the absolute bare minimum to not get into trouble.

7. More than adequate.

8. Good. Even if near-perfect, few Dutch people would rate above this number because we don't want you to get all cocky.

9. Excellent. In service, only awarded when you did something completely unexpected or memorable.

10. Perfect, flawless.

At my last employer they ran regular eNPS surveys using this 1-10 response scale where only 9 or 10 were considered positive. After much complaining from me that this was batshit crazy (I phrased this more appropriately at work), they finally switched to using just three possible responses. The question was something like "Would you recommend an acquaintance or friend with the appropriate skills to apply for a job at $Company?". The three responses were "No", "Maybe", and "Yes".

Fun related fact about 1-5 survey responses in Germany: make sure you specify that 5 is the top mark and 1 is the bottom, otherwise you’re liable to get some really odd results.

This is because 1 is the highest grade (mark) in school, and 5 is failing, as is the even-worse 6.

And even so informed, or accustomed to 1-5 scale used by everyone else, Germans are more likely than Americans to give a 4 instead of a 5 to something that they were happy with, but was not the best they’d ever experienced.

I've noticed that many decent German TV productions (I'm talking about somewhat positive outliers here, like the best few Tatort episodes in a year) get scores of 6 to 7 on IMDB - especially the ones that are only shown in Germany. So, the Netherlands and Germany are pretty similar about that.

> When I lived in America, it seemed the consensus of anything under a 9 is killing puppies.

I also get the impression that American grades are like this:

- A is good. You made an effort.

- A [however modified with +/++]: great job!

- C is terrible. You skipped all classes or fell into a coma.

The insiders say never to rent from AirBnB if the property isn't at least 4.8 stars. These days those ranking systems don't truly start from 0 or 1. The statistically significant range is much smaller. I don't personally know have the stats chops to do it, but I'm sure determining that range can be done.

The important thing is to actually read the reviews. A five-star place that's located 2 blocks away from a railroad crossing that's active all night won't yield a five-star experience, but the host can hardly be expected to mention that.

The last AirBnB I rented was a five-star house where the host had installed one of those Nest smoke detectors with the motion-sensing night light directly above the bed. He still got a five-star rating, but I went well out of my way to mention the night light in my review.

Why 5 if your sleep was miserable (I assume)? Host has full control over that and should be held responsible if made a poor choice which affects guests. You're polite to host, but not polite to future guests.

It was easy enough to fix by taping a washcloth loosely over the sensor, and obviously my opinion regarding it being a terrible idea isn't universally shared, or he wouldn't have installed it there in the first place. So I didn't feel good about dropping a score near 100% to one near 80% just for that one thing.

There was also a massive foam rubber pad on the mattress that made the first night a rough one... but again, some people who are wrong prefer those, and again, it was easy enough to remove. Or at least, it would've been if my partner hadn't already been asleep on it when I came to bed. The resulting tossing and turning was what made the motion-sensing night light a problem.

Basically I didn't want to go full Karen on the landlord because everything else about the house was comfortable and thoughtful. I felt that a four-star rating would have been projecting a combination of my personal preferences and comically-bad luck on his other potential renters.

It would be interesting to find out the mean and standard deviation for various review sites, and write a browser extension that normalizes them.

Yeah in Australia anything under a 5 out 5 may seem like grounds for a defamation lawsuit.

It's a complete farce.

I very often go to places that are 3/5. I was burned too often bad 4 or 5 places. The more important favtor is number of reviews and what exactly recent reviews say.

About skewed ratings:

0) there is no objective point of reference, every rating is about the reviewer more than the place.

1) google knows about it. they have weighed rating of some sort so 2/5 rating is not going to kill.

2) people in some cultures can give low rating without explaining what they didn't like. either it is complicated to explain what it is or it may be misinterpreted or not publicly acceptable.

I disagree that stars should be evenly distributed between 1 to 5 stars. I think it's quite possible that most books that people choose to read end up being a 3 (good with some flaws) or 4 (good but not all-time great). It's kind of like the same thing with pizza. I'd give most pizza a 3 or 4. Very few 1s and 5s to be sure. 1 doesn't have to mean bottom 20% of pizzas. It can mean "awful, couldn't finish," where very few pizzas would fall into that category. 5 doesn't need to mean "best 20% of pizzas," it can mean "telling strangers about it the next day," again where very few pizzas would fall into that category.

I had a debate on the Criticker site about this topic since they try hard to turn your raw ratings into a normalized span. The fact is that because of ratings sites, I'm very rarely watching 1-star movies. That's good! So the low end was dominated by a 2.5/3 type ratings, which per my scale meant "just OK," but they were mapping it to "bad," somehow inferring that my rating range is 3-5.

“Normal distribution” doesn’t mean evenly distributed. Most being in the middle is a normal distribution.

The Modern Zelda reviews are a prime example of this.

The non-industry people reviewing Zelda have 4 options, 7/10, 8/10, 9/10, 10/10.

A 7/10 means the game was bad and you had a hard time finishing it, and won't be playing again.

8/10 means the game was also bad, but you had fun for a few minutes/hours.

9/10 means the game meet minimum expectations.

10/10 means you enjoyed the game, but there were countless flaws that took away from enjoying the game.

10/10 Greatest game of all time means, the game was above average.

Now, this same system applied outside Nintendo's curve, you knock these numbers down 2.

Its incredibly hard to figure out if a Nintendo game is good, if you treat them like other companies. Even Nintendo believes this and will yank early access/ads/etc... from gaming websites if you don't comply with the Nintendo curve.

Devils advocate; Is it not possible that Nintendo just makes good games?

They spend a lot of time on development, don't rush things, the trailers are honest about gameplay and what to expect.

I can understand it's not your cup of tea or arguments that their online service is shit (it is) but reviewing a video game in isolation, many of them seem to be good games objectively. Games that don't appeal to the average CoD fan sure, but that doesn't mean they are bad games.

Nah, you can see it on subreddits as people discuss the game. Complaining about a few enemies, copypaste world, etc... Its not a 10/10, but it will still be reviewed like this.

Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then?

Don't get me wrong, I religiously play all the zelda's and find ways to enjoy them. I am under Nintendo's spell, but I also know these are the corporate mascots I grew up with. Nintendo markets to children, we basically need therapy if we want to be free from their grip.

That’s fair. I would take a minor issue with your framing (someone who didn’t play BotW wouldn’t find the map a cop out and I’m glad they put the effort in for the 2 other layers of the map) but you could make those same arguments against COD or Destiny.

I do think it’s worth noting how many of the ideas from BotW have been copied in the past 6 years in other games.

I guess my point is, I can understand the 10/10s but I also find it entirely reasonable to view it closer to a 7 or 8.

We’re also not thinking about things in a vacuum in this discussion. Is the Xbox game released as if BotW never happened? Nintendo certainly uses its brand to buy marketing influence but importantly, I don’t think the company is pissed when games fail.

Skyward Sword was a good example of that actually. It’s failure literally led to BotW. Nintendo got off their laurels and reinvented their “tried and true” formula and now it’s paying off.

I am closer your position for games like 2D Mario. Zero innovation and I’ll be eagerly watching the reviews for the new game later this year.

> Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then?

You're right, but to be fair sort of 'reskin and re-rate' test would decimate the reviews of most sequels, remakes, adaptations, etc. Not just in video games, but movies, books, etc. Would Peter Jackson's The Lord of the Rings trilogy have been half as culturally relevant if it weren't standing on the shoulders of a very popular series of books?

> Complaining about a few enemies, copypaste world, etc.

It's important to review a game as it exists, and not a game that you imagined in your head that would be better.

> Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then?

Well, in the case of Genshin Impact: lots and lots of money.

Seriously, though. If Breath of the Wild or Tears of the Kingdom had been released by a company other than Nintendo, people would be singing its praises to the high heavens. You'd see a lot of "Why can't Nintendo come up with this" type articles about it.

> The non-industry people reviewing Zelda have 4 options, 7/10, 8/10, 9/10, 10/10.

Did you mean the industry people? Plenty of user reviews are 1/10. It's the industry guys that are always rating stuff 7+

A simple point that address a lot of that is just that bad games usually get fewer reviews. There are still plenty of 6/10 and below games getting reviewed, but (contrary to the idiom) you can often accurately get a decent gauge of media ahead of time. People want to experience good media, and so better media tends to have a gravity of critic and audience attention. Most reviews don't go below 7 because reviewers are people who, just like the rest of us, want to enjoy good media.

A couple examples: Lord of the Rings Gollum (a recent, high-profile, but poorly received game) has 36 critic reviews on metacritic for an average of 39, Forspoken (another similarly high-profile and poorly-received game) has 7 for an average of 63, and Tears of the Kingdom has 139 for an average of 96. Someone with more free time and motivation could pull a better spread of data to really show how the two data points relate, but I'd be shocked if there weren't a strong correlation between score and review count almost across the board.

> the average book should have a 2.5 Disagree on this. This assumes that people read books completely at random. In reality, people read blurbs/summaries or get a recommendation and the typical book you read is likely to be better than the midpoint between the worst book you've read and the best book you've read (I'm assuming a linear scale mental model).

> This assumes that people read books completely at random.

I'm not sure it implies that. I think that if:

1. Book quality follows a normal distribution.

and

2. People are capable of accurately assessing the quality of books relative to other books.

then you'd end up with an average rating of 2.5.

It doesn't matter if people are disproportionately reading the better books and rating them highly because an increasing number of reviews on a book would only give you more confidence in its rating, not change the rating of the book relative to other books.

To illustrate this we can imagine a world with only 100 books in it and only 100 people in it.

Let's assume that a single person from that group reads all 100 books and per the two assumptions above correctly rates them on a normal distribution.

Then the other 99 people read only a single book which the first person recommends as a 5. And per the two assumptions above they all also rate it as a 5.

Now there are 99 books with only 1 rating normally distributed between 1 and 5 and there is 1 book with 100 ratings - all 5s.

Does this change the average rating of books on the platform?

Nope the average rating is still 2.5. In fact the one book could have a million ratings or a billion 5 star ratings and the average rating of books on the platform would still be a 2.5.

What we see on sites like goodreads is that either book quality doesn't follow a normal distribution and/or people are not capable of accurately assessing the quality of books relative to other books.

Good points. But do we have evidence GR is not a normal distribution. GP’s comment does apply to views of ratings, so we might have an impression if higher average rating because the books we look at are more often better than avg.

You missed #3 all books get rated.

Terrible books likely have fewer readers, thus fewer and likely zero people would post a review.

IIRC they initially gave you a text for each star like

1 - I hated it

2 - It was okay

3 - I liked it

4 - It was great

5 - I loved it

I thought it was helpful for framing what a score would be but I haven’t seen this in the app or anywhere for a long time. At least it avoids the problem of Amazon where a review is like “Best book I’ve ever read but the copy I received had a rip on the dust jacket - 1/5”.

Learned through experience to skip anything below a 4 or very high 3 now. Probably missing some interesting stuff but I don’t read that often anyway.

See https://www.criticker.com/explain/ -- it works sort of like what you're suggesting. You rank movies from 0-100, which is different from rating them. Your percentile ranking scores are compared to other users' rankings, and then it can suggest movies that it thinks you will rank highly.

IMO the real problem with something as subjective as books or movies is that even completely honest, well-reasoned reviews are going to be all over the map. My review of a Pride and Prejudice movie is going to be maybe 3/10, but my wife would give it 7/10, while we have the opposite reactions to something like The Hunt for Red October.

I don't care about reviews from experts or the unwashed masses. I don't even really care about reviews that much -- I'm more interesting in ratings from people who like the same kind of stuff I do.

There's also TrueSkill, by Microsoft Research, which turns Elo into a proper generative model: https://en.wikipedia.org/wiki/TrueSkill

It'd be definitely interesting to see a book review site like this. In the meantime, for math books, I trust curated rankings by MAA: https://www.maa.org/press/maa-reviews

We use this at https://languageroadmap.com, but for difficulty rankings between titles in the context of language learning using media. It does seem to work pretty well and we disclaim rankings with the confidence score.

As for the grandparent comment: recency bias, as pointed out by another commenter is a thing, as is the tediousness of doing a bunch of pairwise decisions. I think a happier medium is to have everyone fill in tier charts (with variable number of tiers) and build the pairwise rankings from that.

> I just give you two books that you've read and you tell me which is better. Do this loads of times and eventually you have a solid ranking of every book you've ever read. Aggregate everyone's rankings and you have a much more robust system then "please rate this book out of 5 stars".

We've done exactly this with Flickchart - https://www.flickchart.com - presenting the user with two movies at a time and letting them choose what, in their opinion, is the better film. We then aggregate everyone's rankings to generate a combined chart of the best films of all-time that's always in flux and changing.

We take into account each user's total number of rankings (refinement), and the total number of movies they've ranked (breadth of knowledge). We also use time as a factor - given that something that just came out has had less time to be "an all-timer" than something that has existed in the world longer and had the time to be experienced by more people.

We created this way back in 2009 and are currently building the next-generation of it on a modern web stack. This new iteration is also more modularized so we can expand more easily from just movies to other domains (books, games, music, TV, etc.).

And in many such systems the peer pressure to give a 5 is tremendous, which further devalues the rating system. Uber drivers for example have told me that if more than a few customers give them a 4 they will be effectively fired. Restaurants, tour companies etc have asked me to please give them the maximum number of stars on Yelp, Google, Tripadvisor, etc lest there be dire consequences for them.

This is of course Goodheart's Law in action but I don't have any ideas for fixing the problem.

Elo/Trueskill sounds like a solid idea.

When you go to rate a book the system asks you if it was better or worse (maybe same) then a book of you previously read closest in rating or some hidden matchmaking rating (MMR).

I think having books having both a personal and global score would be nice as well.

I don't think it would be possible for someone to change their rating but that may be a feature.

I also don't think this solves recommendation though it may be useful as a feature.

> I don't think it would be possible for someone to change their rating but that may be a feature.

It should definitely be possible. Take films. I had a friend who would go see a film. On the day he saw it: "OMG THE BEST THING EVER". Next Week: "Yeah, that was a good film". A month later: "Yeah it was okay I guess". Six months later: "A bit ordinary really"

In 1990 if you'd asked me to rank the Alien films, I would have put ALIENS first, and ALIEN second. These days, I reverse those rankings.

People's tastes change over time. It would be inaccurate to "fix" a rating in stone.

Yeah, I think what was meant is that the Elo rating system might make it difficult to account for changes in the rating.

People change their minds all the time (and it's a good thing), I agree.

Yes, this is what I meant. Changing the rating would require storing every rating and recalculating all dependent ratings which could be quite a large DAG.

While not impossible I don't think it would be particularly scalable.

Well rating scales don't have any absolute meaning, only that which is attributed to them either by the reviewer or the audience. On any aggregate review site the scale of the aggregate score will take on its own semantics, which is the aggregate of all the users' own semantics they use for their individual scores.

And for most people, it seems like the semantics of rating scales don't align with a normal distribution about the 50% mark, but something more like letter grades where 70% is decent/mediocre, while 50% is a failure (in the US).

IMO this makes at least as much sense as centering around 50%; if you only score 50% of the points on a test I don't think you're competent in a subject, and if your book only meets 50% of the criteria for a great book then it might be a bad book. But again, it's all arbitrary and only makes sense in the context of the particular reviewer or aggregate community.

It's not just because humans are bad at distributions. I would rate almost all the books I read at least 3/5 because I pick my books carefully. There are plenty of terrible books deserving 1/5 but they are often obvious from a distance.

With eg movies a much greater investment is required to tell if it's terrible.

I really like the idea of relative ratings.

But that also amplifies a problem that exists with any rating: recency bias, or any sort of time bias.

I change over time. An album that I loved when I was 15 could get a 1/5 from me today. Or a book that I hated in my teens, may now be my favorite. It's complicated.

In which case the system would need to continually challenge you about your historical choices, perhaps based on recent choices and detecting possible paradoxes (you liked A more than B, and you liked B more than C, but you also liked C more than A).

Very good solution indeed!

Maybe also allow ratings to be a list over time rather than one, single data point. That'll make calculating an average more involved, but also more realistic.

For a while now I've though that maybe only three rankings are needed:

* Recommend this thing

* Neutral opinion

* Do not recommend this thing

Not sure how much "neutral" would be used: kind of like thumbs-up/down or up-down-vote mechanic: if you're neutral you may simply not vote at all.

I just give you two books that you've read and you tell me which is better.

I played around with a tool for sorting through my personal photos based on that idea.

It gets interesting when you start thinking of how to optimally choose pairs to compare, to maximise signal and minimize redundant comparisons. This becomes more important as the size of the set of objects you are comparing becomes large.

https://github.com/tim-fan/image_sorting

Yes, one person I knew rated my book 2/5 because, as he said, it was beginner/mid-level-oriented and he wasn't aware when he purchased it (clearly stated on the book's page, and pretty much everywhere). A star rating system gives more emphasis on an individual's own preference of what a good rating should be which works against the consumer.

> I just give you two books that you've read and you tell me which is better.

I've always been so for this in situations where the users also have a database of their own history on the site, like Goodreads or Boardgamegeek.com, for example. Just throw up modals every once and a while comparing two books that you know the user knows, and make it easy to opt out of. I probably wouldn't opt out, because it would appeal to my ego and feel like a game.

> Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage.

Just normalize the users against themselves. I wouldn't be giving a 4-star review, I'd be giving a pessimizer(4-star review).

As much as the IMDB rankings are obviously gamed (especially when a movie or series just came out) I find the ratings fairly accurate over time, a 7+ is usually pretty decent to great for 8+, ratings of 6 are usually pretty average and 5 and below are truly awful.

Agree. Except for Indian movies. Their ratings aren’t trustworthy. Lots of 8+ movies they’re mediocre or bad (despite high production costs).

This reminds me of my favourite scene from the social network when movie Zuck gets the idea for facemash:

Billy Olson’s sitting here and had the idea of putting some of these next to pictures of farm animals and have people vote on who’s hotter. Yea, it’s on. I’m not gonna do the farm animals but I like the idea of comparing two people together. It gives the whole thing a very “Turing” feel since people’s ratings of the pictures will be more implicit than, say, choosing a number to represent each person’s hotness like they do on hotornot.com. The first thing we’re going to need is a lot of pictures.

Agree completely. Scoring is mostly a cultural thing, I presume based on how the person experienced grades in school and their social pressures to attain certain grades. As a child I moved from a country where getting any grade below 9/10 was a shame to one where 7 was good enough, a 9 was very rare.

Because of this all the review systems like IMDB are flawed by design.

Steam reviews are the most trustworthy for me. While they are not perfect, the binary choice normalizes these scoring differences, and can be read as a "probability that you will enjoy X if your are interested in this sort of thing" which is the best one can hope for.

First only IMDb TV shows hover around 8, films hover around 6.

And it is still useful. For example, from my perspective I can quickly judge a movie from an imdb score.

Anything less than 6.1 is bad. 6.1-6.3, ok, 6.3-6.8 good, 6.8+ excellent.

Apart from movies starring star trek characters, add one to all those scores.

Goodreads is similar but tighter, anything less than 3.9 is a red flag, over 4.2 and you start getting into excellent.

Though in my opinion steam seems to have got it done best with "overwhelminly positive", "very positive", "positive", "mixed", "mostly negative", etc. but occasionally gets review bombed.

> I just give you two books that you've read and you tell me which is better.

I can't wait to compare The Pickwick Papers against The Unfolding of Language.

A solid point: books may excel at one dimension while not even attempting another. I suppose one might have to ascertain what genre, exactly, a book falls in, and how it fares in its own genre. But then you're not comparing pairwise, and we lose the benefits of our visual cortex or whatever neural system is so good at pairwise comparisons.

Constraining pairings within their genres might suffice, if possible. There are several standard ontology for books, and it's not a trivial problem.

I don't think it makes much sense to compare a fiction book to a non-fiction book. There's not enough points of comparison. While both can be compared on quality of prose, the fiction might be judged on characters and plot, and the non-fiction book on accuracy and detail.

You might as well ask the question: Who is the better Tennis player? Roger Federer or Garry Kasparov?

A system would need to draw some kind of boundary conditions so you are comparing things that have more in common than "printed on paper and bound".

I don't think I'm alone in that if I'm reading something that is anywhere near a 1-star, I don't finish it. And then go on to not leave a review. This skews my ratings indeed, because not every book I start is between 3-5 stars, but almost every book I _finish_ is.

This article about Microsoft's research on recommendation algorithms was interesting. It's from several years ago, so I imagine some people have begun using it. https://news.mit.edu/2017/better-recommendation-algorithm-12...

> Humans just aren't very good at ranking things on a normal distribution

Anecdotally this feels correct, the more common user rating distributions I see are an inverted bell curve - lots of 5's and 1's, not too many 3's.

But as far as the same problem that plagues every other review system goes, I would say the paid/fake reviews are the far bigger problem.

I don't find the average rating very useful. I do, however, find reading 1 and 2 star reviews quite helpful.

It wouldn't be very helpful to read a review as to why someone thought Fundamental Calculus was better than Moby Dick. Without the star rating system, you lose the ability to filter on the reviews from people who think the product is bad.

Couldn't you do this implicitly if I rate book A an 8 and book B a 7 then I have technically said book A is better than book B, so ignore the numerical value and take the better reading. Now put the rating out of 100 as out of 5 or 10 would lead to too many ties.

> For IMDB the rankings all hover around 8ish.

That's definitely not the case. 8 is neither mean nor median for titles on imdb. If a film scores anywhere above 8.0 they are already in the top 250. Most titles average between 5 and 7.

I have thought about this relative rating as well. Would it be possible to use? I mean, does it scale? I also have seen some ratings where you can only rate between 5 - 10, I guess since no one ever rates below 5 anyway. I think it kind of works.

It seems like it would be easy to just normalize scores. Rather than a book’s score being the average rating they’ve received, it should be a z-score based on all ratings.

In medicine they use a pain scale with faces when they need to find out from kids how much pain they are in. Something like that might be a better system?

> aggregated personal Elo

Interestingly that seems reminiscient of data apps. What are your thoughts on their ranking philosophy?

> Humans

There's the problem there – assuming human input makes up an important part of the rankings.

Reply on news.ycombinator.com