Hacker News Reader

Comment by tyingq | Hacker News Reader

Not immoral or illegal, but using human raters to seed an algorithm can make the results worse. For example, it could skew to favoring websites where the rater is familiar with the brand. Which might rank mediocre branded content over less well known sites with much higher quality content.

That seems to match what I've seen over the last 10 years with Google.

View on HN

I worked for Leapforce for a while doing this sort of rating work using Google's guidelines. The guidelines were linked elsewhere in this thread and, at a glance, they are the same ones I used:

https://static.googleusercontent.com/media/guidelines.raterh...

If you're doing something that would skew the results to favour a certain brand, you'd be going against Google's guidelines and your contract would be terminated quickly. I found the review process of my results to be fairly stringent; they were reviewed frequently to ensure I was staying within the guidelines, and I was told even about slight deviatiations.

I'm not suggesting it would be on purpose. Trust must have some subconscious effect.

Can you give some examples of higher-quality-content sites that Google search is consistently missing?

Examine.com is one, here's the prior HN thread. Had it not been for that HN thread I would never have known that site even existed, despite having frequently searched for queries where it would have been extremely relevant: https://news.ycombinator.com/item?id=20676755

examine.com is the first organic search result I get for "supplemental vitamin d sleep". Maybe the results are hit-or-miss depending on the subtopic (or Google re-tuned the algorithm in the intervening 3 months).

> Not immoral or illegal, but using human raters to seed an algorithm can make the results worse

...versus asking an oracle what the "good" results are? It's not clear what you'd propose as an alternative.

Mitigating controls, expecting the bias.

At the moment it's a Frankenstein set up with a rules/algorithm based ranking subsequently shuffled around by several separate ML runs with different aims.

The FTC did once unearth a Google attempt to purposefully influence the human ratings: http://graphics.wsj.com/google-ftc-report/ (long read, but worth it). Find this section of the paper:

"Unable to get positive reviews from raters when Google demoted comparison shopping sites, Google changed the raters' criteria to try to get positive results."

Reply on news.ycombinator.com