An exploration of Yelp's own filtered reviews
jamiehdavidson.blogspot.com
jamiehdavidson.blogspot.com
- When you post a review, you as a reviewer think its unfiltered forever. When you revisit the page as a logged in user and read a place that has your review, your review is visible. When you log out or log in as another user, the review is filtered and hidden. At the very least, it should tell you your review is filtered, I see no reason to pretend the review is not filtered when the review is legitimate.
- When you view unfiltered results, the per page number mysteriously changes to 10 per page. I don't see any reason why this should change. Plus the results are pretty slow to load, quite slower than the results for filtered reviews.
- Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews? I don't see the difference, unless they want to prevent people from writing scripts to pull in unfiltered review data. Plus the captcha is fucking horrible, literally half the captcha's I get are not readable and I need to refresh.
- The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered. For example, take this user: http://www.yelp.com/user_details?userid=tZlbsUVo-8wtnR7oMa-3... . The guy has 11 reviews, 1 1-star review, 1 2-star review and nothing out of the ordinary and yet his review about Yelp was filtered. Why? His points in the review seemed legitimate. He seems to be a normal user, not a new user and posts reviews across the board (more good reviews than bad in fact). They should either fix the algorithm or be more transparent about why reviews are filtered because I can't understand why a review like that is filtered.
Maybe they don't want Google to see them...
I can explain that for you! If you're gaming the system and Yelp catches you, they don't want to tell you they've caught you, or you'll just try again. That's also why you need to enter in a captcha to view the unfiltered reviews -- they want to prevent automatic methods of confirming that posts were filtered.
Apparently this isn't common knowledge -- the OP makes the same mistake:
"Interestingly enough, you have to pass a reCaptcha when clicking on it. Weird, I don't have to do that to view unfiltered reviews. Interesting...."
EDIT: ps, the user you linked to doesn't have a name. I'm not saying his stuff should be filtered, but keep in mind when you're looking at this stuff that a million factors go into that decision, and it's probably in their customers' best interests to err on the side of filtering a review.
This only affects people who have written legitimate reviews, that have been filtered. It does nothing to prevent an automated system from doing anything.
You need to find more cases like this. Just one data point is not enough to draw any useful patterns.
Caching, I'm sure most unfiltered reviews are cached whereas filtered reviews are not and reaching out past the cache can be expensive. One way to mitigate this is to reduce the number of results you pull.
> Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews?
If you can write a script to deduce the filtering algorithm then you can by definition write reviews that thwart it. With less data, it is harder to deduce the filtering algorithm. In other words, a captcha thwarts high-volume review fraud.
> The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered.
I think most people seem to underestimate the difficulty of the problem. Unlike e-mail spam, which is easy for a human to spot, fake reviews are very hard for a human to spot. How can you tell if a consumer was provoked into writing a positive review so that they could get a few bucks off their order just from their writing? You can't, you can look at other statistical trends behind such reviews (such as a sudden wave of positive reviews), but you're only looking for side effects of the primary problem and thus you will never achieve perfect performance from a method like this.
Yelp takes the (somewhat philosophical) viewpoint that customers who are coerced into writing a review are less genuine than they would be otherwise. I believe that this view drives a lot of their algorithm and possibly threatens its accuracy in a way that is ultimately not worth it. I think there are a number of things that Yelp could do to make the users trust in reviews greater that don't involve filtering - one simple thing would be for a user's review of an Indian restaurant to show me that user's breakdown of reviews of other Indian restaurants.
TL;DR: This is a much harder problem than it seems at first glance, partly because of the nature of the problem and partly how Yelp has framed it for themselves.
Disclaimer: I used to work at Yelp, but no longer do. Everyone I worked with were stand-up guys.
But they clearly don't want users to see unfiltered reviews. A tiny gray link below all 40 reviews, then a captcha (or two or three) and then a slow user experience before you can see the filtered reviews is lame.
I agree with you about the showing other reviews of the same subject, that would be neat. I guess if I were Yelp, I would try harder at standing up for their algorithms and show more data about why they work and why we are better off having their amazing algorithms.
I had an experience a year or so ago with a friend who started a moving service in SF. A couple of months after he started the business, he noticed he received a review on Yelp from some dude that said during a moving job, the guy took a smoke break and peed all over the sofa he was moving. Not only was the story ridiculously false but my buddy had no idea who the reviewer was. The review did however NOT get filtered, even after he responded to the review and contacted Yelp. And he was stuck with this crazy review at the top of his profile. This went on for months and it really damaged his credibility, meanwhile he would have positive reviews from legitimate customers who would naturally have a newer profile or whatever and the reviews would get filtered. It just seems like Yelp should be more sophisticated. (And yes, they are 10000% better than TripAdvisor)
(I hope this comment makes it through the filter, I swear it's not a fake... I don't even know newhouseb...)
The author seems to think people hate Yelp, but I'm not sure that's the case. Everyone I know in real life uses it regularly when trying to find some place to go, and the results are largely acceptable.
Review sites are always going to trend negative because people who have had average or good experiences aren't going to be driven by rage to write a nasty review. Everyone has their own star scale and expectations of service ("I had to sit in economy class on my $10 ticket! I'm never flying United again!"). This leads to useless star ratings, but this is no fault of Yelp itself. It's the fault of relying on non-professionals to do professional-quality work. But, if you read for content, you can usually figure out whether a place is good or not. For example, a review like "I went during the dinner rush on Saturday night and it took 5 minutes to get a table! 1 star! Oh yeah, the food was good." is a positive review, even though the reviewer only gave one star.
So anyway, don't hate on Yelp, hate on the clueless people clueless writing reviews.
Anyway, the evidence is clear for there to be good reason to hate on Yelp (see my article). But like you said, there's also good reason to hate on clueless reviewers. Why can't we do both until either issue is fixed?
You claim that Yelp is writing their own 5 star reviews:
Personally, I wouldn't be shocked if Yelp is putting in its own 5 star reviews for itself in order to offset the barrage of negative reviews. Common perception of Yelp is not good, far from their current 3-star rating. Just glancing over the first few 4 and 5-star reviews for Yelp, all but two came from an 'Elite' member. Of course they review Yelp highly, they get thrown crazy parties. And I wouldn't be shocked if you lose your Elite status for a negative Yelp review.
This is pure FUD. You don't know if you lose elite status for a negative review. Not everyone gets to go to crazy parties. If this were Wikipedia, every statement would be tagged [weasel words] or [citation needed]. You've just made a bunch of stuff up.
If this is just entertainment, fine, but if you think you're writing something useful that is worthy of HN, then you're a little off. Your article boils down to links to filtered reviews not being obvious enough (even though the link is the same as all the other links on the page), complaints about reCaptcha, which Yelp doesn't even control, and then a bunch of made-up facts/anecdotes that you have done no research to prove true or false.
Look, I don't even like Yelp. But this is simply sloppy journalism, and it reads like it was written by a fifth grader who got a bunch of one-star reviews on Yelp.
Notice I said "I wouldn't be shocked" twice, not "I know for a fact that". I not once said that was the way it was, I simply said I thought it to be a decent possibility.
And this isn't worthy of HN? It reads like it was written by a fifth grader? It's number one on HN right now. Did I do something to you at one time in the past? If so, I sincerely apologize for ever wronging you.
Do you really think your article is an example of top-quality writing and analysis? (Also, why post your own work to social news sites if you can't take the criticism? I don't hate you, I hate this article.)
And no, I definitely don't think my post was top-quality writing or analysis. It was my second blog post, so I'm far from those labels. However, I'd place it a notch above a fifth-grader.
I can't explain their filtering algorithm because I didn't write their filtering algorithm. Ultimately, automatic fraud assessment is the only way Yelp is going to scale; it's too expensive to have a human validate every review, so you get the computer to do it for you. 99% of the time, it works. 1% of the time, you get the situation where the dog training company gets screwed. It's like when your credit card gets flagged for buying gas out of state or using an ATM in China. It's annoying that you got false-positived for fraud, but that's what happens. It's the only way the credit-card system can work. Same for Yelp's filtering; no automatic filtering, no Yelp.
The problem with Yelp is that people don't take it with a grain of salt. But that's not Yelp's problem, that's a "people are morons" problem.
I personally more inclined to hop on Yelp as a warning to other consumers on a poor business, vs, just random "ups" to other well run businesses. I gain no benefit but altruistic feelings, and this, takes time. The time it takes to post a negative review is rewarding in itself to try and "help" others avoid or be wary of said business.
Also, markets should take a factor in analysis of Yelp reviews. As an example, larger cities have a much higher preponderance of reviews as a whole, where even medium (what I would call them) such as SLC (where I live) are very sparse and therefore limited in search / decision making.
That original article says that the users banded together to write positive reviews over a short period of time. We don't know whether the business owner incentivized them or if they just did that altruistically. The facts are simply not known, the best we can do is make assumptions there (which makes for a pretty weak argument to begin with). Banding together and writing reviews with the sole purpose of boosting a business' online reputation is what really happened here, and Yelp specifically designed their algorithm to deal with those use cases. The actual pattern businesses follow for building a reputation is very incremental: positive reviews spread over time, not a short burst by people who aren't actively using the site to review other businesses.
To make a compelling case, rather than what appears as a rant in its current form, you need more data. You need to show that the users are active on Yelp and they aren't afraid to write both positive and negative reviews to reflect their true experiences. Banding together in groups to write positive reviews is basically the same thing as submitting a story to HN and getting all your friends to upvote it.
That shouldn't be necessary. I've written like, five reviews ever and they're all honest, I have a few friends that have written reviews and they seem to average one review every few years or so. Meanwhile I've seen reviews from elites that are well-written but absolutely pointless for me. Sometimes it's as if they're writing more to impress other reviewers than to actually give me any useful information.
Banding together and writing reviews with the sole purpose of boosting a business' online reputation is what really happened here, and Yelp specifically designed their algorithm to deal with those use cases.
Why filter the reviews entirely instead of just sorting them down or something? Especially if they seem more or less legit except there's a small bunch all at once?
Banding together to boost an online business's reputation really should not be cause for filtration. Fake accounts, bots, lies, and attempts to bury negative information are problems. A bunch of loyal customers who hear about yelp from the owner and want to write a review is quite simply not a problem and yelp shouldn't be doing anything about it.
And why discriminate against businesses that don't sell advertising?
How do you algorithmically distinguish between the two? To Yelp, someone who signs up for an account and writes one positive review is the exact same thing as a script written to create bots that go off and write one positive review. They're both identical. And the bots' behavior can change to write more reviews to prevent looking like a bot. Business reputations take time, so when Yelp's algorithm sees "this business has been around since 1975, and 100% of its positive reviews just came in over the past 10 months, where 100% of its reviews were previously negative before that", that should be a red flag. I can't find the age of Frank's dog training business on Yelp; I searched and I didn't see it listed. So I can't say for certain whether the age of his business on Yelp had to do with the algorithm flagging the reviews.
And why discriminate against businesses that don't sell advertising?
Yelp is letting you bump up your flagged reviews if you pay an advertising fee. If Yelp's algorithms don't trigger any red flags, positive reviews should show up unfiltered on your page whether you pay for advertising or not.
Maybe it can't. It doesn't matter you use a linear or binary search if the program isn't fed valid data.
I'm suggesting they are using an algorithm to solve a problem that it shouldn't be used to solve.
And the bots' behavior can change to write more reviews to prevent looking like a bot. Business reputations take time, so when Yelp's algorithm sees "this business has been around since 1975, and 100% of its positive reviews just came in over the past 10 months, where 100% of its reviews were previously negative before that", that should be a red flag.
Interesting that you use the term "red flag" since that usually means a "warning" or "needs attention." There's a world of difference between using an algorithm to raise red flags, and using an algorithm to automatically prune content. In fact, a three-decades old small business with a lone, 1-star review should raise a red flag about the 1-star review.
I'm inclined to agree with you about the original poster's perspective but it is hard for me to get past the serious accusations and evidence presented by numerous business owners; that Yelp basically extorted them when they received negative reviews and were a new listing on Yelp.
Based on multiple reports it appears as if Yelp's business intelligence group actively targeted(s) businesses that met certain criteria (new, very few reviews), sent those results to an inside sales team, and then if the customer didn't purchase from the inside sales rep Yelp effectively changed their filter process for that individual business to "penalize" them for not paying to play.
Zagat, Urbanspoon, and TripAdvisor, have never to my knowledge ever implemented these type tactics. I might be wrong, but barring a set of insurmountable complete coincidences for each and every one of the businesses that reported these algorithm changes in the way reviews were displayed, I've got to conclude that Yelp is using their inside sales teams to pressure (extort?) businesses that their business intel teams identify, and make them pay dearly if they don't pay.
A more likely explanation of "extortion" is that the sales org hires 21 year olds fresh out of school who routinely don't make it past 90 days. I'd imagine some of these reps might resort to shady practices to fill a quota, but that's also why they don't last past 90 days. I think this is just a problem of needing a sales organization to make money.
Disclaimer: I used to work on the engineering team at Yelp, and I have the utmost respect for the intelligence and morality of the engineers on the review filtering team.
https://biz.yelp.com/support/common_questions
It's easy to jump on the bandwagon and say "Yelp is evil," but having been a near-constant point of controversy, it's something they do their best to address.
They are accused of hiding positive reviews for businesses that don't pay them. You can perfectly well write this logic in Python for an filtering algorithm:
def filter(review):
...
if review.business.contacted_by_sales_count >= 3 and
not review.business.paying_yelp and
review.stars >= 4 and
random() > 0.8: # 80% chance
return true # yes, filter it
...
Disclaimer: I used to work on the Yelp engineering team, at one point on the Salesforce data refresh project.
You can often make it through after painfully screwing up the hard to read word.
If you don't like the reCaptcha ($deity know why, but it's your right), complain to the website, don't fuck it up for everyone else.
If half the things suggested about Yelp are true, they risk their integrity at great peril.
If business have a good rating on Yelp, then that pretty much mean that business is ok. However, a bad rating might be because business was not nice with Yelp sales people.
That is ok because they target small, unestablished, businesses which you will never discover that they are actually excellent business because they have just one star review.
In other words, advertising at Yelp does not guarantee you a good review. However, not being nice with their sales people might get you a bad review if you are a small business.
And this is absolutely nothing new.
and judicial system and all the yelp users accept it ..?