Hacker News Reader

Comment by dragonwriter | Hacker News Reader

> Unless, they fix this and get more people to review honestly. I think they're going to run into a lot of problems.

I don't airBnB, but a friend who does was recently berated by a host after leaving a 4-star review; that host claimed (don't know how true this is) that many hosts will refuse to rent to you again if you leave anything but a 5-star review.

That and other things I've seen and heard about airBnB reviews suggest to me that there a structural problems that are likely to make it problematic for them to become honest and useful in general, even on top of the usual problems in numeric/star rated reviews that come from cultural differences in how people rate when given such a scale even when the actual substantive opinion is the same.

View on HN

AirBnB reviews are massively inflated. A 4.5 average means you should carefully read between the lines of what the previous guests wrote. A 4 average means you should probably reconsider staying. Hades itself would probably rank around 3.5.

I think a better tell would be displaying the ratio of people who actually reviewed vs the ones who didn't. When people don't like a place they'll usually not review it at all.

That's a great point that I forgot to mention! I will only book a place with sparce reviews (under 5 or so) if the host is very new (listing is less than a month or two old).

I recently booked a place in an apartment complex, and when I arrived, there were big signs at all the entrances saying it wasn't a hotel, that it was for residents only, and anybody found to be renting daily/weekly would be arrested as trespassers. I was only staying there for a couple of nights and didn't have time for the hassle of arguing with Airbnb at the time, so I just took the risk.

After my stay, I reported it to Airbnb. I didn't want to leave a review because I didn't want to deal with the fallout if Airbnb banned the host and they knew I was responsible. Airbnb ignored the report, the listing is still active, and none of the reviews mention the problem. I wonder how many other guests reported the problem to Airbnb without leaving a review as well.

Best bet is to report it to the condo association or property management company. AirBNB will basically turn a blind eye to those kind of rentals.

That's right. Airbnb is the Uber of hotels. One place I stayed, I reported to the hoa and my host basically threatened me until I had to block his number.

Until airbnb starts cleaning up their mess I think people will eventually realize that this is just another scam in the long run.

I had a similar experience (apparently these signs are common in Airbnb apartments in Thailand).

At the same time, the apartment front desk staff and doorman were completely aware of my staying there temporarily. I showed up, asked for the keys, the front desk had an envelope, and there were no questions. They must have someone doing that every few weeks, and the signs are someone else's business.

I live in condo in Bangkok where a couple of units are being rented out on Airbnb despite the warning signs.

Most of the inhabitants are people with 9-5 jobs who wish to enjoy a quiet evening after a long day at work. Having tourists next door doesn't help.

People coming home drunk late at night, kids playing soccer in the hallway, Scandinavian women walking around topless and in thong at the pool* - Airbnb guests treat the condo building like a resort, which it isn't.

The walls and doors are thinner, there's no carpet in the hallway and there are less elevators than in a hotel.

Here's a message to everyone considering renting an apartment on Airbnb: GET A GODDAMN HOTEL ROOM!

Thank you SV for "disrupting" my quiet evening [/endrant]

* I know some might actually enjoy this, but Thailand is a conservative Buddhist country. Respect that please.

People actually review. Otherwise you won't be able to see what the other person wrote about you.

I know that's what they say in their reminder emails, but it's very misleading. Basically, people can't write a review after they have seen the review of themselves. The mechanism is designed to prevent retaliatory reviews.

So you get a window of a few weeks in which you can write a review. If you leave a review, then you get to see the review about yourself straight away. But if you don't review, then you have to wait until your own window has closed. After that, you're able to see the review about yourself regardless of whether you left a review yourself – because you are no longer able to leave a retaliatory review.

Airbnb aren't very clear about this at all, and that certainly pushes people to leave reviews, but it's a side effect that Airbnb play up, not the actual designed purpose of the review system.

That's not true. You can see what the other party wrote without writing a review if you can wait for the review period to end, which is 14 days -- not that long. It must be so because reviews are public.

I haven't done eBay in ages but rating inflation used to be a big deal there when the relationship between buyers and sellers was more symmetrical (among other differences from today). People took anything other than a positive with an "A+++++++++++++++++++++ seller" as meaning that the seller had shipped you a brick and then burned down your house. People got really pissed off if someone left a neutral because, say, the item was damaged in shipment even if the problem was ultimately resolved.

That episode of Black Mirror is coming to mind for me with this discussion. Once the currency of your profile rating points affects your livelihood all sorts of bizarre social ramifications start to rear their heads. The difference between a 4.5 out of 5 and a 5 out of 5 becomes huge.

The whole system of review curation that infests online markets is a problem waiting to be solved. Right now we basically have to parse reviews on the 4.4-5.0 scale because anything outside that is meaningless.

It's amazing to go read a novel rated 3 stars (because it operates outside this loop) and find it transcendently good.

It's a difficult problem. Survey response scales have probably been a matter of research and discussion before the web even existed.

There are particular instances (like in this case) where there are a lot of incentives for just about everyone to give 5 stars, 10 stars, or a positive and call it a day.

But, in general, there are issues with self-selection, bias toward the high end of the scale, ratings meaning different things for different types and classes of product, etc.

Reviews are a problem, but not for novels.

Perception of books and concepts covered within are entirely subjective.

However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.

> However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.

No, all assessments of value are subjective (and for most services involve subjective weighting of multiple different subjective quality factors.) And the mapping from an internal value judgement to a star rating is something which is a separate subjective evaluation, and one on which research has shown substantial cultural variation along ethnic and other lines.

While that may be true for other people, I have no trouble giving a purchased object a rating for its objective value (but I do excessive research on buying options beforehand). Even if the judgment is relative to other products, it can still be objective.

The rating may be inaccurate because you can only measure in full stars, but it'll still be objective and the inaccuracy is the fault of the rating system, not me.

Objective (adjective):

> (of a person or their judgment) not influenced by personal feelings or opinions in considering and representing facts.

>Reviews are a problem, but not for novels.

Hmm. I'm willing to accept that the average reader is a more sophisticated and thoughtful consumer than for some other product categories. And there may be less overt tribalism and fanboyism in book reviews.

But, in my experience, there's still a lot of people who will almost unthinkingly give 5 stars to a favorite author or genre. (And I know some of those people who are pretty uncritical about SF books or whatever.)

Goodreads reviews are still pretty inflated. Anything around or under 3.5 or so probably has some big flaws.

I have found no correlation between the Goodreads score and my own perception of the book.

Again, novels and products are entirely different in that you're hopefully shopping for a product by its specs. You can't 'research' novels beforehand (unless you count as reading other novels by the same author as research) and they don't have any specs to go by.

Instead of asking customers to give numeric star ratings, perhaps a better option would be a short list of Yes / No / Unsure questions. Were there clean sheets on the bed? Were there any cockroaches? Did the appliances work? Etc. Such questions are more objective and less prone to inflation.

They also require more time, and most people don't bother to leave reviews anyway.

But if you're going to review, a few extra seconds shouldn't matter

It's how the "official" reviews from Marriott/Hilton/etc all work

1. Require each person to answer one specific question per listing. Yes/no/almost

2. Ask each person a different objective question, intentionally filling in the gaps.

3. After a few people, you have an objective, relatively thorough view of the house, but you've not burdened any of your users with more than one question per stay.

Booking.com does it, btw. And I strongly prefer it.

Agree - it really helps you identify which parts of the experience are problematic. If the location is 8.7 where everything else is 9.5+, you know you'll be walking a distance to get anywhere useful. If the location is 9.5 while cleanliness is 7.8, you can consider the trade-off to be in the thick of things.

Surprised there is only one mention of Booking.com here. It is an indispensable tool for booking my vacations. The reviews are very useful and a lot more accurate than things like yelp. I have glanced at other websites, but nothing gives me the reassurance of the legitimacy of the reviews that booking.com does.

So do low end hostel sites, funnily enough. Arguably they need it more to distinguish between the places that are popular because they're clean, good value and quiet and the places that are popular because they have a very good (and loud) bar and a vendor who's very good at persuading people to go on his tours

The new review flow on Airbnb asks for more details about specific aspects of a listing.

This is probably a much better way to do it. One of the food delivery company does it this way (I think it's grubhub/seamless). Three questions before you're given an option to review: 1. Was the food on time? 2. Was the order correct? 3. Was the food good? Only then is the person given the option to review.

The problem with star based rating systems is that they're INCREDIBLY subjective. I oftentimes read yelp reviews that are 3 star reviews where the person raves about the restaurant and then complains about a minor issue. I have a similar issue with app store ratings where I've had a rash of people rate one of my apps 1 star with the review text reading "I just downloaded this, I'll raise my review to 5 stars if love this app".

Also a lot of times on Amazon the low reviews are reflective of an issue with that particular sale instead of the item itself (ie damaged while shipping, etc).

Another annoyance is amazon not allowing rating separate editions of something. For example a bad kindle version resulted in bad ratings for Dune a few years ago. Sometimes certain versions of movies are bad while the movie in general is great.

Is anyone trying to solve this right now? It seems like a really interesting problem that would be really useful to solve.

It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback.

Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?

Many services have this issue, that perfection is the minimum requirement. I don't understand why, given this, we don't just go to a thumbs up / thumbs down rating system. We have one option that means "fine" and four (or nine) that mean "terrible, and a risk to your business".

I know I personally gave plenty of three star reviews on Lyft before I learned better, meaning them as "no complaints". Who knows what happened to those drivers as a result. Why even invite the misunderstanding?

This could be a cultural design thing. A lot of these apps (Airbnb, Lyft, etc) are from the US, where getting full marks in school is a frequent occurence if you study reasonably well.

In France for instance, it's very rare for a student to go above 15/20 or so at the college level (I graduated in the top 10% of my class with a 12/20 average; the top ranked student, whom I was friends with, had a 14/20 average).

But when I graded homework as a graduate student for an american university, the professor told me I was too harsh, and after adjusting my criteria to match what she wanted, I found myself giving many 100/100.

The US is also where you insult service staff and deprive them of income by giving them a gratuity which would be considered generous thanks in most tipping cultures. But at least that's baked into one's understanding of the price.

When it comes to ratings used by others to evaluate, no system is more customer and vendor-hostile than one where everything that meets a minimum threshold for adequacy should be rated as perfect.

It's silly that they have a system that pressures you into lying.

In a lot of ways, yes. On the other hand, it seems like a hard problem to solve - how do you get honest feedback in situations like this, where you want to use it to both improve things overall and judge your employees/contractors? Maybe you can't.

I guess, at least a small step would be to make it so values less than the highest rating don't count against you so negatively - it should be ok to be a 3/5 or 7/10.

I wouldn't even believe the salesman that anything other than 10 will get them into trouble. I'd just refuse filling it out.

Has anyone ever tried

1. normalize reviews to make a 4 star from someone who always gives 5 stars worth less than a 3 star from someone who always gives 2 stars; and

2. make reviews secret?

I've wondered this myself. Someone with a dozen 5-star reviews is essentially providing no information and should be removed from the equation.

How about making you rank the places you've stayed at? You could do it by subset if there's a lot.

Or normalize the ranges between min/avg/max of the ratings given by each individual user to min/middle/max of the scale before merging the number into the total.

If you do it secretly like shadowbanning and introduce it softly by slowly blending between conventional score and normalized, it would drive a whole generation of SEO/social/share-economy experts delightfully mad. Think of all the free publicity!

Jokes aside, it might help to regard low ratings as what they are: a way to flag bad behavior, dead serious stuff. We should stop pretending that it is a game, then maybe solutions will be more likely to appear.

This doesn't seem to be as big of a problem in Japan.

On the popular restaurant review site Tabelog, a 3.6 (out of 5) is a very good ranking; there's more 3-star Michelin restaurants in Tokyo than restaurants above 4.5.

https://tabelog.com/en

Fundamental challenge with star rating system is your perception of what counts as 5 star & my perception of what counts as 5 can be very different. It basically comes down to what your personal standards of something 5 star are. I am generally a very adjusting person, so minor glitches might not lead me to move from 5 to 2 star but someone could be more particular than me in which case his 5 star will move to 1 star at slightest issue.

I had the same thing. Stayed at a really disgustingly dirty place. No toilet paper, no bed sheets, no curtains on the windows. Gave it a bad review and had the host rip me apart about it. "What the fuck did you expect? I would feed you grapes?".

>I don't airBnB, but a friend who does was recently berated by a host after leaving a 4-star review; that host claimed (don't know how true this is) that many hosts will refuse to rent to you again if you leave anything but a 5-star review.

Airbnb pushes hosts really, really hard to achieve >80% 5-star reviews. If <80% then the listing gets pushed down the rankings and you lose SuperHost status.

Some guests never give 5-star reviews for whatever reason. They really, seriously hurt the host. If your average is >4.0 (and it should be over 4.5) then any review less than 5 stars will lower your rating.

I worked in a restaurant and was trying hard to get our TripAdvisor ranking higher. One quiet night a guy comes in so we chat and give him some great service, special dishes etc. At the end of the meal he said it was excellent and had left us a 4 star review. I said "Dude, our average is 4.8 stars, you just made us look worse."

In the end we got to #2 in our city. The #1 was a cafe that was only open 7 am to 4 pm. All you need for 5 star review in a cafe is a hot coffee, so we settled for #2.

But AirBnB reviews don't work like that. Even though you leave star rating, nobody sees it (I don't know how it looks like on the host side though, as I only used it as a tenant). Only feedback traceable to you is text you write, but you can be honest with stars. And you see compound rating on the property page, which they only show when there's enough reviews (so again, if you left your review first, they can't see what it was either). I think the system is pretty good.

The review system for AirBnB has gotten more complicated over the years.

https://qz.com/410264/you-should-never-trust-an-airbnb-revie...

I have been an Airbnb host for a while and if you follow your reviews closely, you can tell when someone leaves you low star ratings. In the host dashboard, you only see aggregate stars, but it will break it down as x 5-stars, y-4 stars, etc for each category, so if you know that before your last guest left a review you had 20 5-stars and 3 4-stars and now you have 4 4-stars it's pretty easy to figure out where that came from.

I'm a host as well, and their new stats section breaks down the reviews in detail. You can see who left what stars. Don't know if this works the same for the traveler.

> cultural differences in how people rate

This is an interesting problem, and yes, needs to be accommodated some how. I used to manage a SW product and service offering that had customers around the planet. We sent out quarterly report cards in order to ensure we were meeting customers needs. It was As and Bs exclusively except for one country that was Ds and Fs for the same level of quality and service.

Some type of calibration or break down is needed.

In the Silicon Valley world where Trump is the Silicon Valley candidate except for ideology, that calibration would be manipulating everything to average to 4.5 stars while appearing random.

Isn't that why AirBnB is pushing the Instant Reservation system really hard?

Nah, that's mostly to reduce the long tail of rentals.. in some cases it takes months to turn a person into a customer and book now/instant reservation helps shorten this cycle and reduce off platform fishers/scammers.

And it helps address the racism issues they've been having trouble with.

this is why Uber punishes drivers who go under a 4 star average. 5 should be the default.

Well don't go there again then? - AND lower the score and report the user if those are possible.

Reply on news.ycombinator.com