SF Restaurant Sign Calls Yelp ‘Bully,’ Integrity of Reviews Questioned | Hacker News Reader

SF Restaurant Sign Calls Yelp ‘Bully,’ Integrity of Reviews Questioned

foodbeast.com

89 pointscobrausn87 comments

The problem with these claims is that they all confuse ignorance with malice. These rumors are persistent because they make a sexy story, but they aren't true -- Yelp doesn't manipulate reviews based on advertising.

Citing what a Yelp sales rep is alleged to have said to a business owner sounds like compelling evidence, but it's really just garbage: Yelp sales reps have no ability to influence anything, anywhere, no matter what they say or don't say. They're part of a huge team doing a low-pay cold-calling gig -- they make a sale, ring a little bell, and hand in a piece of paper documenting the sale for the bean-counters. That's the extent of their influence with anyone.

What's really going on here is some combination of business owners mis-interpreting what a sales rep says, businesses trying to transparently game the system (and whining when it doesn't work -- business owners are amazingly stupid about this!), sales reps saying dumb things (I'm sure this happens occasionally, despite best efforts to control what they say), and the somewhat random nature of an automated review filter. There are tens of thousands of businesses being called by Yelp sales reps every day. Even if 1% of those businesses have reviews randomly filtered on the same day, that's a lot of opportunities for people to see patterns where they don't exist.

I infer from your comments that there would be no way to prove based on review display that Yelp was providing review relief if you pay for advertising, right? You basically have to trust their algo?

The algorithm isn't published, but you can click "filtered reviews" on the bottom of the page and eyeball them yourself / decide for yourself

Which last I looked was super hidden and required a captcha to even read.

The CAPTCHA is probably to prevent people from scraping unfiltered reviews and using unfiltered reviews to attempt to reverse engineer the algorithm.

To your first question - By all means, you could prove it. Establish relationships with and systematically monitor the displayed and filtered reviews of a statistically significant sample of businesses. Record when they are contacted by Yelp sales reps, record their decisions on those calls, and classify their filtered review behavior before and after. This is doable, and it would be enough information to determine if there is actually a correlation in review filter activity to advertising offers.

Go for it. As a former Yelp engineer, I can assure you that the answer would be negative (no correlation), but I invite you to give it a shot.

To your second question, the answer is "yes". A spam fighter simply cannot publicly release their spam detection algorithm - it would make the system too easy to bypass. SEO black hats have been trying for a decade to get around Google's sorting algorithms, and then they cry foul when Google changes things up. The message from Google all along has been: Stop trying to game our system, and instead focus on creating good websites. Yelp's situation is no different.

2 other data points from (disclosure) someone with ties to Yelp:

1. Ongoing independent research / A working HBS paper found no connection between advertising and filtering (see last page of: http://hvrd.me/ZNpuOy) [Why do we see advertisers are more likely to have their positive reviews filtered?]

2. A simple Google test like this: http://bit.ly/15QV1WS [Why are these advertisers not using their 'Delete negative review' button? Hint: because it doesn't exist. Which begs question -- If it doesn't exist, is there a rational incentive to sell it?]

So sales reps say that your good reviews will stay and your bad reviews will disappear, but what they say is untrue?

I don't know about the Thai restaurant owner, but I certainly trust JWZ. http://www.jwz.org/blog/2009/02/yelp-shakedown/

OK, you trust JWZ, but did you read what he wrote, or did you just glance at it and make a snap judgment because you think he's cool? Here's what he says in easy-to-consume bullet points for the TL;DR set:

1) The linked post is four years old. Yelp used to sell a "featured review", but hasn't done that for years because it was controversial (duh). So that part is true, but no longer relevant [1].

2) Stupid people with bad taste write Yelp reviews. This is definitely true (hello, internet), but again, irrelevant [2].

3) He links to the tired old East Bay Express article that started the whole "Yelp extortion" meme, and which has been definitively debunked. Repeatedly [3].

So, fine. Trust JWZ. He's a good, smart guy. But in this case, you're "trusting" a mix of his opinion (which is valid), and out-dated, second-hand information (which is not).

[1] http://www.smallbusinesssem.com/yelp-kills-featured-reviews/...

[2] http://www.youtube.com/watch?v=QEdXhH97Z7E

[3] http://online.wsj.com/article/SB1000142405297020450530457700...

So.... you're saying that it's a bad idea to trust what people say about businesses on the internet, because it's often outdated and lacking detail?

What does Yelp do again? I forget.

It's bad to indiscriminately trust what completely random people say. That's different from not trusting at all. There's a reason why Yelp displays friends' reviews before other random reviews, and why Yelp goes to significant effort to get people to write textual reviews rather than just leaving a star rating.

Trust him about what? That article says: - He's "pretty sure" (direct quote) that they offered to take down reviews. Hardly a smoking gun. - He doesn't agree with the reviews of his business. No surprises there. - There's an article in a newspaper where a bunch of business owners claimed Yelp punished them for not advertising. - I think he even implies that Yelp's PR people being scared about that article is somehow evidence of it being true...which is borderline funny.

I trust my heroes too, but like, that blog post doesn't really...say anything.

A little off-topic, but this remark by JWZ is funny:

> I can't stand Yelp. Mostly because it's always the top hit when I search for a restaurant instead of that restaurant's own web site, which is useless, but also because I wouldn't trust any of its reviews for a second.

Yelp's SERP ranking is actually their best feature...because they at least have the important facts of the store (its phone number, address, category, etc) in a predictable, usually parseable format. JWZ's post is from four years ago, which means that many more restaurants were using Flash (many of them still do now)...Yelp's listings overtaking those was a welcome phenomenon for me

You can't just brush off sales reps claiming to be able to improve reviews. If Yelp isn't paying them enough and getting crap, that's Yelp's fault. If Yelp has out-sourced his to someone else, they have out-sourced their reputation.[1]

[1] http://www.popehat.com/2011/12/27/outsource-your-marketing-o...

Yelp's cold-calling operation is in-house.

It's naive to think that you can control what a team of thousands of sales people say while dialing for dollars. No matter what you do to control the script, there's going to be someone who breaks ranks to gain advantage.

It's naive to think that you won't be help accountable for what the people you pay to represent you are saying, too.

I agree with most of what you say, but this is vastly overstated: "There are tens of thousands of businesses being called by Yelp sales reps every day."

If they call 30K businesses a day, they would call every single business in the US in 3 years.

http://www.census.gov/econ/smallbus.html

Considering that they're trying to sell to every business in the US, 3 years actually seems like a very long time.

They are not trying to sell to every business in America. Do you think all businesses are restaurants and barber shops?

Do you think yelp reviews are only for restaurants and barber shops?

My business makes apps. Is Yelp going to call me soon?

I use to be an account exec at Yelp, we had a goal of calling 300 businesses per day. I don't know how big the sales team is now, but we had a class of 30. And there would be a new class every month. And that was just in the SF office. They have a whole sales team in AZ and NY. 30k doesn't seem far fetched.

If they called one business every 10 minutes, they would need 625 full time employees to pull that off. What kind of business has 625 employees? And why are they only averaging one call every 10 minutes? Impossible, I say:)

I've actually heard from other business owners also threatened by Yelp salespeople after not accepting their offer to increase ratings. Except, in the cases I've heard, the Yelp salesperson actually implicitly threatened a decrease in ratings if the business did not buy ads.

Regardless of whether these salespeople have the power to do so or not, it's not entirely ethical business practice. And this leads to business owners unaware of the review filtering algorithm who have encountered these shady salespeople to try to find correlation between the two and end up pointing fingers at Yelp.

So I wouldn't say Yelp is free of blame in these situations, they should be more readily firing these salespeople and shouldn't have a culture that encourages their salespeople to act this way.

Yelp is an extortion racket.

This is a recurring story that does not go away. Once is an accident, twice is coincidence, third time is enemy action.

Further, I judge people by their actions, not their intent. "Oh, you didn't mean to shoot me in the face? No problem!"

My two heresy anecdotes...

My buddy has a restaurant. He's always playing cat and mouse with the Yelp reviews. He claims Yelp buries the positive reviews and that he's been told they would reorder the reviews, for a price.

More worrisome is my doctor's story. I checked my doctor's Yelp review. Doctor is trashed. Which surprised me, because this doctor is great. So I ask about it. Doctor claims reviewer was a disgruntled patient looking for pain medication for migraines, which is something this doctor (a neurologist) does not do. So doctor contacts Yelp. Sorry, sir, there's nothing we can do. Perhaps you'd like to buy some advertising, we can adjust the results.

Straight up extortion.

The plural of anecdote is not data.

What level of proof do you require?

Something more reliable than hearing "from a totally trusted individual, why would they lie?"

People repeat untrue things a million times a day for varied reasons. In this case, people who don't read up on Yelp's FAQs are frustrated by the yelp filter on both a consumer and business-level and want to subvert them, so they regurgitate a story they heard from someone else as fact.

An email or bill for review promotion/deletion services rendered would be perfectly acceptable, otherwise less credulous persons will file them away with the rest of the bitter cranks.

I'm on the other side of the fence. Walks like a duck and all that.

Without checks and balances and transparency, it's easier to assume the fix is in. The mere appearance of a conflict of interest is sufficient to signify a conflict of interest.

Credulous, huh? That's a good one.

"The mere appearance of a conflict of interest is sufficient to signify a conflict of interest."

And what gives there a conflict of interest? The algorithm has absolutely nothing to do with what advertising businesses pay for or do not pay for. This much is known.

Yelp is not Consumer Reports.

Yelp, Google, UrbanSpoon, YellowPages all charge for placement. And that's fine. But the veneer of collaborative filtering is a con. It's no different than the industry standard quid pro quo system of reviews for ads in tech / product magazines.

Yelp's claim to some form of algorithm objectivity doesn't change the power dynamic.

In other words, I don't credulously believe there to be a functioning Chinese firewall separating the business functions within Yelp. Because those firewalls rarely work, if ever.

But hey, to each their own.

"This is a recurring story that does not go away."

Same with any of the urban legends on Snopes.

A conversation I had with Friend 1, a successful business owner, and Friend 2, a less successful business owner:

Friend 1: Have you tried getting some positive reviews on the internet? People really trust them, it really helps to drive business.

Friend 2: Oh, really? How many reviews do you have?

Friend 1: About twenty.

Friend 2: How many of them did you write yourself?

Friend 1: (Laughs) About nineteen of them.

I'm not surprised by this. Writing reviews or mentions yourself or getting others to astroturf for your product is essential to getting started nowadays.

Sadly it's essential because everyone does it. So when you have no reviews as a new business you look really suspicious because you're outside of an artificial norm.

False, and it's opinions like these that let people justify their own dishonesty.

This is one of the good things about social networks like Facebook. A review itself doesn't matter unless you know the reviewer.

If you build a business model that knowingly showcases the worst ignorance of an anonymous Internet public, and ask for money to help mitigate the inevitable damage, how is that any different from a protection racket?

The algorithm also exists to protect a business from the worst ignorance of an anonymous internet public, it requires participation to avoid being filtered.

If the claims are true, it's a big deal. Losing half a star on Yelp can translate into 20 percent fewer people coming to your restaurant on a busy Friday night, according to a study by two UC Berkeley economists: http://www.dailycal.org/2012/09/06/online-customer-reviews-a...

The article says that probability of a restaurant being sold out at peak hours changes by 20 percent, not number of people.

This is not the first, second, or third time such allegations have been raised, but they always devolve into he-said/she-said.

Restauranteurs, when you're calling Yelp to ask about missing reviews, RECORD THE CONVERSATION! One recording of a Yelp rep saying something like that is game over.

One recording of a Yelp rep saying something like that is one instance of a sales guy lying to make a sale so he can get a commission or make a quota. You know, something that happens all the time.

Yelp could never manipulate reviews and this restaurant owner could have been told exactly what he claims. The two things aren't mutually incompatible.

So who is responsible for what Yelp's representatives say? Are they just a force of nature?

Yelp, obviously, but 'greedy lying sales guy' is a different class of problem than 'company-wide extortion scheme'.

Just remember that California is a two-party consent state when it comes to wiretapping- if the other party does not know and understand that you are recording the conversation, it will not be admissible in court proceedings.

Maybe not in court proceedings, but definitely will affect the court of public opinion.

It's not merely an issue of admissibility. It's also a crime to record a call without the other party's consent...

Chiming in to remind people that Oregon is a one-party consent state -- which means so long as you say the conversation is being recorded at the beginning, everything is good. This does not go across state lines, however.

Yelp doesn't have sales offices in Oregon. They do in California and Arizona.

Yelp records calls itself which is why it easily wins its legal matters.

An important distinction here - the allegations don't just devolve into he-said/she-said: they've been outright dismissed, with prejudice, by courts of law as without merit.

Yelp has tens of thousands of paying business customers, and yet with all the sensationalistic media play this type of story somehow continues to generate, no one has ever produced empirical evidence of a single documented case of pay-for-play. I'll also point out that there are at least three former Yelp engineers on this thread - some of whom didn't even like working at the company - who are steadfastly defending Yelp's business practices as legitimate and above-board.

By all means, record it if it happens. But don't hold your breath.

"Yelp’s reviewing process, noting that the company favors consistent reviewers over first timers, in part, to avoid “fake” reviews"

Well, this is how it works more or less

I've never had a comment on Yelp taken down

Well, as other comment says, the problem is the 'idiots'.

There's always someone that thinks McD is the epitome of gourmet food and thinks anything that's not overspiced/overMSGd is "bland"

There's always someone that thinks that a place with tacky decoration and a queue in front and rude bouncers is "the place to be"

So it's really like herding cats.

Cats that love writing reviews that trash a place they don't like, regardless of how quirky their preferences are.

You want to see how the masses think just watch American Idol or a similar show.

I write bad reviews sometimes, but usually include concrete reasons why the place sucks in my review. I find the bad reviews generally are more specific than the good reviews.

And that's totally ok, after all, not all places are great.

But it's one thing to write "OMG this place sux", another is to be specific as you say, like "we waited for 10 minutes until the server appeared and he was rude and the food was chewy"

Only if you're stuck treating people as one homogenous glob.

If you let people self-identify by interests (+french food, -foodie, +fine-dining) you can easily get actionable recommendations and minimize noise. The trick is that adding specificity exposes how small your user base is. And when you're dealing with an inherently local industry, even a massive user-base can start to look small when it's sliced up geographically and then by interest.

This is an interesting idea, especially if it doesn't depend on 'self picking' alone but also on history. (for example if you reviewed 10 burger joints you're probably an 'expert' on it)

Looking at their yelp page, they do have 5 star reviews, and they also have 1 star reviews. There are few filtered reviews, but they aren't universally high ratings or anything, and I think Yelp does filter out users with a single review that is 5 stars because they are usually fake.

I'm also sure that a restaurant feels that recent reviews should be weighted heavily, almost exclusively, and that those two star reviews from last year should be ignored.

> I'm also sure that a restaurant feels that recent reviews should be weighted heavily, almost exclusively, and that those two star reviews from last year should be ignored.

I agree 100% with that. It's very possible for a restaurant to get better or go downhill in a year.

If a restaurant has a rough opening but drastically improves after a few months, the early 1-star reviews can do some serious long-term damage even if they're a year old.

I agree -- recency is more important IMO. Particularly reflecting change in management, change in processes, etc.

For restaurants that aren't chains, processes may not be tried and tested -- first week jitters, hell, first month/quarter jitters should be accounted for.

Link to referenced article: http://www.latimes.com/news/local/la-me-banks-yelp-20130420,...

It seems that Yelp is turning into something like the BBB, ultimately holding the good name of organizations hostage, whether or not that was the original intent.

I think that's a little unfair. Yes, there have been some instances of that happening with the BBB... but when one of their sales people contacted me to sign up to become a BBB accredited business, I asked if it would affect my rating negatively if I refused, and the sales person was very adamant that it would not. So I declined, but she updated my profile with how long I was in business (adding an extra 3 years), and that improved my rating from A- to A+. Which is my company's current rating, even though I have never given them a dime.. all I do is respond to customers who file complaints with them.

Technically speaking, there's nothing to stop Yelp from manipulating results within their search engine, just like there's nothing to stop Google.

I mean we all think Google doesn't mess with search rankings but we all know they do (see the google blog post about catching Microsoft by inserting fake long-tail search results).

In short, does Yelp do this? I don't see why not; there's basically no way for anyone to verify whether Yelp does this unless they get to dive into emails for discovery in a big trial. Then again, I have to think Yelp has enough integrity not to mess with results; but my point is you would never know except through hearsay and gossip.

There's no ChillingEffects for Yelp Reviews.

For the record - I used to be a an Engineering Manager at Yelp, and I've had my Yelp work emails searched in discovery during big lawsuits about the review filter. So did all of my colleagues. The result of all that should speak for itself.

How is this any different than Google selling ads and deranking spam?

I think the reasoning is that showing ads, and marking them as such, is just fine. Bumping scores that are marked as customer reviews just because the business is buying ads would not be fine. This is not the first time that someone has accused Yelp of doing the latter in addition to the former.

The ads Google sells are marked as advertising. Spam isn't; it's usually not too hard to tell, but it mucks up the search result space.

"Featured reviews" for paying customers essentially become a form of paid advertising, but they are not clearly marked as such. That puts them closer to the spam side than the ad side.

Yelp killed off featured reviews a while ago, specifically because they were controversial. It'd be useful if people didn't keep harping on things that aren't even valid anymore.

Don't know if the accusations are true but I would take Yelp reviews with many grains of salt. Its not the truth I question rather its the extreme subjectivity.

Some people care about service, some don't; some care about decor, some don't. Different people have different tastes as well as different price sensitivity.

Yelp reviewers are also self selecting both in who is more prone to write reviews and under which circumstances that will drive someone to write reviews.

So if you really want to figure out how good a place is for you, by all means use Yelp as a resource. But you need to review the reviewers to see what their leanings and expertise are and then read their reviews with that understanding in mind.

"Its not the truth I question rather its the extreme subjectivity."

How do you solve this problem? It's endemic to user-generated content. Of course reviews of everything aren't objective fact. They're inherently skewed by opinion by even the most professional paid reviewer.

Down for me now, but the Coral Cache works:

http://foodbeast.com.nyud.net/content/2013/04/22/sf-restaura...

While I've seen the filtered reviews issue and think the claims are somewhat true, my bigger problem is that yelp does nothing for reviewer quality.

So many times, I've seen blatantly false and misleading information tacked on to someone's reviews. The site allows both the microcomments "Food, okay. Waitresses, hot." and the long, trailing rants about how the steak tasted rotten. This is a problem, and not one easily fixed. Being an "Elite Reviewer" just means you give out more idiotic reviews than the rest.

Yelp is a joke to anyone who really cares about food.

"my bigger problem is that yelp does nothing for reviewer quality"

What do you expect from user-generated content?

"Yelp is a joke to anyone who really cares about food."

It's really not. You take the reviews into aggregate, it's very easy to screen out the idiots and fickle eaters. Don't pay attention to the ratings, pay attention to the depictions in the reviews.

This might be a common problem. Forums that, while ostensibly democratic, are actually tailored to suit the forum-owner's agenda.

It's done by selective promotion, censorship, filtering, etc. "Boing Boing" is one famous culprit if I recall correctly.

The owners argue "it's my forum so I have the right; and besides I'm not censoring I'm unpublishing" or some such nonsense.

I'm sure it can get very sneaky. fake democracy.

Another problem is that people are more compelled to write bad reviews.

Right. I don't see any problem with INITIAL reviews being held in a queue based on the user's participation level. With more participation reviews get unfiltered.

Criticizing Yelp's algorithm is fine, but people don't understand how it actually works, nor do they have the patience to figure it out. Full transparency would be nice, but that would defeat the purpose to having it to screen out the fakers and SEO spammers.

This is why I use Cloudy [http://askcloudy.com/]. They don't do any of this nonsense and it's better to see reviews from friends anyway.

That's useless to me. When I want recommendations it's because I'm somewhere other than where I live and the people I know live.

It will expand to friend of friend if it can't find friends and will show global ratings. It's been good so far for me when I travel

I would like to remind people that this is not the first or even second time that this kind of claim has flaired up regarding biased reviews, corrupted reviews, purchased reviews, and extortionary practices.

I would also caveat though that there is a plausible possibility that this is an attempt at mobbing Yelp.

I have always felt that reviews are ripe for disruption and a bit of an ethics and authority treatment.

"I have always felt that reviews are ripe for disruption and a bit of an ethics and authority treatment."

I'd be fascinated to see how someone tries to tackle this better. I post reviews there sometime and see people arguing against their algorithm, but their (imperfect) algorithm is working against a social problem, not a technical problem and that's where persons get confused.

"Ethics" is a pretty poor word to use when Yelp is working against the equivalent of or direct SEO-style review spammers. It's hard to tell a real person from a plant.

the only thing i use yelp for nowadays is directions and general info like phone number or hours of operations.

looking restaurants / dry cleaners / whatever up in google and/or apple maps is pretty much useless when you're on the go.

whereas on yelp, you simply enter the name (or ethnic category), click on the map, and there you go.

quite frankly i stopped using yelp for reviews as soon as the idiots took over. /hipster

quite frankly i stopped using yelp for reviews

I have too. My preferred method of searching for "good" restaurants is:

1) Viewing reviews on Opentable when making reservations (to discover which dishes are good)

2) Looking at the Zagat rating when clicking on a restaurant in Google maps (which is more analogous to a hotel's star rating and is more objective)

They should remove the captcha to view filtered reviews.

Reply on news.ycombinator.com