Amazon's overcomplicated new product star ratings are no bright idea | Hacker News Reader

Amazon's overcomplicated new product star ratings are no bright idea

androidpolice.com

59 pointsshscs91166 comments

Most reliable metric: ignore 5 star reviews and only compare the number of 4 star reviews to the number of 1 star reviews. this works across all review sites. most sellers only fake 5 star reviews. however, if you review products and services that you know are quite excellent, you will that they often have way more 4 star reviews than one star reviews and more 4 than 3, more three than 2, and more 2 than one. this is because they are consistently excellent. but products with polar review metrics like all 5 star reviews and all 1 star reviews generally i find have the most fraudulent reviews and are most likely to have a bad experience. I think you should weigh 1 star reviews much higher than 5 star reviews. the new amazon metric or the advertisements talking about their aggregate quantity of five star reviews are wrong and misguided.

1 star reviews can also be paid for. I tend to look at 1/2/3-star reviews that seem passionate (was written like a real person with actual emotions - bonus points for photos) and see if the things that bothered them would also bother me.

I like your idea though, and will use it as an extra data point.

I tend to ignore the 1 star reviews because, in my experience, most are either from disgruntled people who wouldn't even try to open the manual first, or absolute binary thinkers.

When it comes to more subjective categories, like books, 1 star reviews are even worse.

However they are likely not fake, or less likely than 5-stars.

Review bombing is absolutely "a thing".

> I tend to ignore the 1 star reviews because, in my experience, most are either from disgruntled people who wouldn't even try to open the manual first, or absolute binary thinkers.

Eh, not always. I was once searching Amazon for a cheap popcorn maker. The cheapest one on page 1 of the Amazon search results had mostly five-star reviews, except for a lone one-star review that said the popcorn make emitted a "tongue of flame" when plugged in.

That was sufficient for me to buy the second cheapest popcorn maker instead.

There's an xkcd for this: https://xkcd.com/937/

Oh, so there is. I was expecting the one about the bobcat.

https://xkcd.com/325/

OT, but for those still in the market, I absolutely love this one:

https://www.amazon.com/gp/product/B002Z9LHAG

Along with the right proportions of Snappy popcorn oil (butter burst tastes slightly more movie-ish but coconut is obv better for you and almost as movie-theater-like) and Flavocol, my popcorn is now way better than the theater popcorn, and I didn't even think that was really possible after years of terrible microwave popcorn.

oils: https://www.amazon.com/gp/product/B003C4UDEY and https://www.amazon.com/gp/product/B00AXK9ROC (we've switched entirely to coconut oil)

salt: https://www.amazon.com/gp/product/B004W8LT10

you can probably get these last two things at commercial supply companies as well (buy local where you can)

These days I just make my popcorn in a covered wok on the stove, honestly. There's probably some argument about the extra fat I'm adding to fry the kernels vs an air popper, but, it works. I get about the same amount of unpopped kernels, and it's one less "unitasker" I have to find room for.

I read the one-star reviews and ask "do these people sound like they're complaining about the product itself, or something else?" (For example, if you're buying books that somebody could conceivably be assigned in school, you'll see one-star reviews that are really reviews of a class they didn't like pretending to be a review of a book.)

True, or my personal favorite useless 1 star reviews:

"The box was damaged [by UPS/FedEx] when it was delivered"

"I didn't know I would have to assemble [this piece of furniture with four screws]"

"This [completely different] product sucks" (though to be fair, this one is probably on the merchant for swapping out products under the hood to preserve ratings)

My favorite of all time though is the "this product does exactly what it should do, but I didn't want it to do that, even though that was what it was advertised to do, and that's the only purpose for this entire product category - one star". E.g. there was one for a l-cystine based dough conditioner, where someone left a one star review claiming that, get this, it made the dough extensible, and they didn't want that because they were making a pie or something. Incidentally, I understand why they don't sell raw l-cystine dough reducer to random consumers, because people would literally kill themselves with it.

Many complaints about shipping damage suggests a packaging problem. And Amazon will close your account if you return too many damaged items.

1 star reviews are often skewed by the "delivery company did something dumb" reviews or a part was missing or there was a problem with returning it or that kind of thing. I often just look that the average rating isn't complete trash, if it's averaging 2 star on a lot of ratings the product probably is not worth considering. Then I pick some longer 4-star reviews and read the text. In the long reviews someone's put some thought into it and they're often more balanced.

This also applies to app ratings, especially for company apps.

Uber's app might be perfect, but the rating will reflect on the company, drivers, and experiences of users.

Anecdotal, I've worked on apps for a train operator and currently a power company, it still holds up.

You can't just compare numbers of reviews. You also have to make sure the reviews are for the actual product.

The practice of switching products after garnering high review numbers is so widespread, Amazon can't pretend it's not tacitly complicit.

Not just that, but sometimes there are completely different products on the same page, and the reviews are shared between them. It's impossible to know the rating of each product in this case.

Also sort by latest reviews. These are more likely to be relevant to the current product and slightly less likely to be fake.

User contributed ratings have, it seems, mostly outlived their usefulness. It isn't that things can't be reviewed, but ratings are like anything else in life, in that they have deaths. You want me to trust ratings more, let me order them by date.

Oddly, Steam does a good job with this. Two ratings, recent and all time. Google, amusingly, only shows the all time on the search page, but the recent reviews are very relevant.

Does this have to apply to all things? Maybe not, but manufacturing changes. needs of the time change. General expectations change. A 4 star rating from a decade is of much less reliability than one today. Even if it is critically a good rating.

My system is to just focus on the 1 & 2 star ratings, and look for trends, specifically. It takes a bit of work, since most of those reviews are just nonsense, but eventually true flaws in the product will be exposed through repetition. Due to the amount of work, I typically only reserve this method for big ticket items that would be difficult to return, such as appliances.

Oddly, appliances are the area I would trust user reviews the least. It will be dominated by luck of experience on who will post. And with how variable manufacturing is for most, the exact same model sold on the other side of the nation is as likely to be a different beast as not. Not even considering last year's version of it.

Still, I don't discard the idea here. If there are features of it that every low star mentions, I think I agree that I would trust the feature is bad. I can't think of often that has happened. Will try to pay more attention for it.

Steam does that well, but its recommended/not recommended system is a dark pattern IMO. People tend to choose recommended even for games they don't like on the basis that someone else might like it, or it might improve in the future. Virtually every popular game has 'Overwhelmingly Positive' or similar. To get useful data you need to read the not recommended reviews in detail.

A more honest system would be 'Bad', 'Average' and 'Fantastic' - then I bet you'd get a ton more 'average' reviews.

Mostly agree, but I wouldn't call it a dark pattern. The problem is you are asking for two things. Do you want popular recommendations? Or do you want critical reviews? Those are very different and very hard to house together.

This is why I mostly view the problem as being with user provided ratings. They basically fall into the same realm as all big data. Tons of noise, little signal, and easy to misinterpret.

I think this highly depends on what is being rated. Non-durable consumer goods, sure, there is very little assurance that what you're getting now is even what you would have gotten ten years ago. Processes and inputs both change. A previously reputable brand might have been purchased by private equity and enshittified on purpose.

On the other hand, if you're reading reviews of Moby Dick, I don't think the perceived quality or relevance of the writing is going to change much on the scale of decades. Maybe it starts to get archaic at some point, but people still get a lot out of reading Homer. If it's something like Principia Mathematica, that might honestly be timeless as long as human-like creatures exist who can read and understand math and physics.

Fair. Books probably are a bit more evergreen on the reviews. Though, I am guessing that is highly variable on the genre? The Steam approach would probably still work rather well. No need to delete old reviews, but having recent ones can still be good to see.

I've found that sorting Amazon reviews by date works pretty well. It isn't uncommon to see all 5 star reviews for a while and mostly 1 star reviews in the more recent history.

Seems like it wouldn't be too hard for Amazon to weight recent reviews more heavily.

It wouldn't take too much effort for amazon to make their marketplace better in general, but they make money off of it being bad.

The incentives aren't aligned, at least until reputation is hurt badly enough to cost them more profit than the short term gains they make by allowing false reviews and false advertising.

They could, but honestly that might be risky. It might just encourage the fake reviewers to go at a slower but more steady pace to beat the system.

> You want me to trust ratings more, let me order them by date.

Interesting idea.

Having been involved with multiple third party sellers on Amazon, my extension of your thought is to have the ability to display a graph of all five rating levels over time.

This makes rating manipulation visible at both ends of the scale.

I know of people who, for example, sent out free samples of their product to a thousand people or more with a suggestion to post a four or five star review (after which they would get more goodies for free).

I also know of sellers who were attacked by either bots or a mob (in one case a competitor hired a team outside the US) who post hundreds of 1-star ratings and negative reviews.

If you could see a ratings graph over time, you should see step changes. This would be an indication of rating manipulation, both on the positive and negative side.

Many years ago I helped the seller who was the target of the fraudulent 1-star scheme prove to Amazon they were being attacked. The attack also consisted of said mob clicking on this seller's ads very early in the morning to exhaust the daily budget. So, bad reviews and your ads don't show-up when people are shopping. They got destroyed within a couple of months.

Graphing a bunch of this activity clearly showed rate-of-change discrepancies. It took about six months for Amazon to finally admit this happened. They had to refund over $100K in advertising that had been consumed through fraudulent clicks. Amazon would no reveal who did this (they claimed they knew, yet refused to provide the information to enable legal action). Sadly, they did not remove any of the fraudulent ratings and reviews.

Every single one of this seller's products absolutely tanked in search, going from pages 1 ~ 3 to 15+. Faced with the reality of the "mafia" behind the scenes when selling on Amazon, they felt they had no choice but to kill the business and focus on areas where open criminal manipulation and attacks were not tacitly allowed and ignored by the very ecommerce platform your business depends on. I know a bunch of people who got really hurt by Amazon through similar circumstances. To a person, most of them were people --not big business-- trying to make a go of selling product online to support their families and aim for a better life. Instead Amazon created an ecosystem that turned into a nightmare and caused most of them to lose money. Some even took out second mortgages on their homes to finance their entrepreneurial venture. Very sad.

At this point, aren't sellers gaming the system enough that the overall rating has become effectively meaningless? Individual reviews, maybe useful. But its not like the olden days where you could scour amazon for good products, ideally you would have a good idea what you want before even visiting the site.

I still find them useful. I use rating volume + ensuring rating is above 4 + checking whether the brand has a legit amazon store/landing page mostly to try and avoid duds. Individual reviews are useful for specific questions. As much as it sucks these days it's still better than going to your local durable goods retailer, having only one option to choose from with no ratings at all.

For me rating volume can be a red flag. Any product that has over 1,000 reviews and isn’t something has been on the market for years, I automatically deduct a star or two in my mind. People just don’t leave legitimate reviews at that kind of rate

It usually means their product is cheap, not terrible enough to get <4 stars, and they paid for a high search position.

I wish this were true but my purchasing experiences tell me otherwise. There's a lot of awful products with a high volume of good reviews. To the point where for common household products, the decision of the Walmart or Target purchaser is usually better than me picking between a dozen generic Chinese products with 4.5 stars. And it's easier!

One hundred percent. Not to mention the possibility of receiving counterfeit goods even when purchasing name brand on Amazon.

There are plenty of items that have been replaced with a different one but maintain the previous product's high review volume. Also there are those with a huge number of fake reviews. I look at the 1-2 star reviews. If they are dumb ones like box was damaged or shipping slow, that is good. If there are 4000 reviews saying "Great product" and 30 saying it went up in smoke with pictures, I am not buying it.

That's my take.

I do my research elsewhere (which has it's own problems), and then check Amazon for specific products (and these days, they often aren't any less expensive for name-brand items).

It doesn't look overcomplicated. When the average goes up, the number of 5 stars ratings (usually) goes up too, but the distinction between a 4.4 and a 4.6 is harder to spot than that between 40% and 60% 5 star rating (in this example, all ratings are either 4 or 5). So you could argue it offers a better discrimination.

But, it also really looks like they want to hide information that would stop you from buying a product. I know the "don't attribute to malice" maxim, but this is Amazon: they know what they're doing. You don't get to be this size by just fumbling around. They are obsessed with the look of their pages, keeping changes minimal, so they've thought about this long and hard. This change does look like a dark pattern.

> From a user experience standpoint, the new rating display system isn’t as glanceable as the five yellow stars. Additionally, the mention of “5-star” and the use of just one yellow star in the new arrangement could mislead new users, at least briefly.

It's overcomplicated enough to have 2 different star indicators. It's an idea that's a few decades old. The late 90s had Bizrate.com managing millions of star images (via java apps) for retail ratings.

Amazon has always been the leading company in the word for rapidly shipping and testing changes. Fumbling around incrementally is their strategy. They made A/B Testing and Multi-Armed Bandit household terms.

Well there is always increased returns and a drop in overall purchases because you believe the rating is bullshit. Maybe they were already so bad on both, that this lifts overall sales long term.

I was so confused by this yesterday. I want to see the number of ratings compared to the average rating. That's how I know if a large number of people have tested this product and reviewed it properly. This system seems designed to hide that. You have to open every product to see that info.

It reminded me how important information design is.

Amazon has an average number of stars there, but it's no longer visually shown - instead, just a number. I have relied on the stars being "filled up" for so long that it now felt like every product was basically the same: a single, lone star. It was so harder to get an at-a-glance feeling of which products were good and which not-so-good.

And unforutnately, that's probably Amazon's goal.

Amazon is nothing but a giant flea market these days.

Everything is 5 stars through fake reviews and everything lies about it's capabilities.

Just today I was looking for a hairdryer and the top one advertised an RPM of 150,000. That'd put its 2 inch turbine spinning at almost 900mph. How much you want to bet the sound barrier isn't broken every time you turn that hairdryer on?

AliExpress with faster shipping.

Honestly I've mostly stopped shopping online entirely. B&M stores have limited space, which means they have to choose what to stock, so they do the work of selecting decent products for me. When I do shop online, I don't use any of the flea markets (Amazon, Newegg, Walmart, etc).

I have shopped online almost exclusively for over 20 years and have started doing the same thing. If I buy a fan at Home Depot, I can be reasonably assured it won't go on fire. They have product buyers that assess the quality and prices of products. Amazon purposely does not have this as it would open them to liability. So "Independent reseller" GOODAIR fan is not at all assessed, and they only provide a platform for users to judge the quality of items and post reviews.

But the GOODAIR fan says it's UL certified and can produce 12047492037 CFM of air!

It has 3,000 5 star reviews, surely it's high quality

Legally speaking, B&M retailers specifically buy things such as UL rated products just because they would incur legal liability for not doing due diligence before stocking and selling goods.

Amazon avoids all of that liability by claiming they are essentially, a flea market.

Hence why Amazon begrudgingly reacted to the FTC demanding the takedown of completely fake smoke alarm listings.

There's also the opposite effect. B&M stores have limited space, and most people prioritize price over everything else, thus most B&M stores thus stock mostly cheap junk that works just enough to keep them out of the court system and keep enough customers happy to keep them in business.

Sadly, if I want a quality item, or a specialty item, and I'm out of luck.

I paid extra to buy drawer slides yesterday from another retailer for the same reason. One of the top sellers on Amazon were some drawer slides advertising as soft close when of course they were not. When the title is misleading, what the hell else is?

I'll pay more to not have to pick through the junk. And I say that as a person who loves to treasure hunt, but I don't want to spend hours reading reviews to find a decent product.

Only 900 mph? That's nothing! Have a look at the flashlights with 100k+ lm brightness. Those are the real deals!

A square meter of the surface of the sun produces 127k lumens. Wouldnt surprise me to see Amazon sellers claiming to be brighter than the sun

There are quite a few who achieve this technical master piece: https://www.amazon.de/s?k=taschenlampe+100000+lumen

It baffles me that Amazon doesn't use their shopper's data more effectively. They could have gone with a weighted average, accounting for number of reviews, consistency of scores, etc. of the shopper in question.

Instead, they want to highlight the most deceptive side of the scores, i.e. 5-star ones.

Kinda feels like they don't want to end fake reviews, but prop them.

I should invent a statistical distribution for scores in 5 star rating systems. There should be peaks at 5 and 1 with a spread in between. Deviations from this distribution would be a fraud indicator, and the 5:1 ratio the rating.

With moves like this Amazon are trending towards the same discoverability quality in their marketplace that Google has achieved with the web as a whole, which may hint at another unknown fundamental law of internet economics.

i agree. i researched reviews for products and services i like and product and services that look fake. i think the top restaurants in my area have a staircase like distribution where 1 star reviews have the lowest aggregate quantity. products and services that suck have more one star reviews than 2, 3 and 4 star reviews and are consistently more polarized with one and five star reviews.

Normalise each person's reviews to make 3/5 their average.

Take notice when people do not leave a review who normally do (hello AirBNB! this is the metric you want!), it often indicates something has gone quite wrong.

I mean five star ratings have always been flawed. Any system where getting 4/5 is bad means the raters are confused, and then it's GIGO. This weird % five-star thing is a workaround for that GIGO because they're treating 4 stars as the same as 1 star.

Personally I'd switch to a "good/flawed/terrible" rating system and just give viewers a pie-chart (or a stacked-bar) of ratings.

Script to add back the review counts:

https://gist.github.com/trickpattyFH20/8d02b18aaf63b7d405456...

Interestingly, I am seeing (on Amazon.com.br) a slightly different test of ”percentage of 4-star ratings or above”

I don’t have an opinion if this is good or bad for customers yet.

I find the indicator for how often an item has been purchased in the last month (on the European site) far more helpful. On a side note, galaxus.ch even shows you the return rate.

Did Q&A also disappeared for everyone at same time ? Its not shown to me for past week or so

I think a sparkline bar chart would be more useful.

That's the old way but it's only on the single item page.

Reply on news.ycombinator.com