'Rater' is paid $10 hourly to teach Google's algorithm

Did you work from an office or remotely? Did you interact with other raters?

Was there an interview process? How about performance reviews? How did Google ensure quality?

1. Do you click on the link in the results page?

2. After clicking on that link, do you go back to the search or not?

3. How long do you stay there before coming back?

Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and found it wasn’t worth keeping.

Correct, SEO bots.

I heard it didn’t work very well for them. The results weren’t very good or helpful.

As someone else pointed out, this pays substantially more than AWS Mechanical Turk. To be honest, it's difficult for me to imagine an easier job than this one. I mean, most people who would do this job could easily now make more at McDonald's or Walmart or whatever. But I think jobs at McDonald's or Walmart are considerably harder, so I don't see why this job should pay the same.

These "Google $title Experts" work as uncompensated customer support (a legal requirement that they treat as such, if the quality is any indicator) on various social media outlets like Reddit and Twitter. They don't just provide free labor, they also go to bat for Google's reputation, acting as marketing and PR liaisons to prevent backlash from users who've been failed by their official support processes.

One 'expert' I spoke with justified working for free by the "connections and access" they received. When I asked if Google would be flying them out to the events they have access to to meet with the people they have connections with, they said they were told they would have to cover their own travel and lodging. I suppose if Google doesn't pay them, they never have to worry about any Experts connecting and starting a union.

We're talking about a multi trillion dollar company. Absolutely ridiculous.

I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something.

Obviously 2005 was a long time ago though and everything could have changed since then.

There is nothing wrong with semi-supervised learning.

They cannot rate all results anyway, so why not get the best possible evaluations?

Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index?

Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?

For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be?

"neovim worktrees"

"django models through"

"pandas econdb"

Standard Leapforce US English raters were paid $13.50/hr. If you were lucky enough to get promoted to "Preferred Agent" (PA), you made $17.40/hr. The criteria for PA promotion was basically to be in the top 5% or so on their quality metrics, plus various subjective quality requirements. Promotion rounds were irregular and random.

Leapforce's US English pay rates never changed over time. There were no raises, other than the aforementioned rare and seemingly luck-based promotion opportunity. I was one of the fortunate ones.

Leapforce raters were misclassified as 1099 contractors. There were no benefits. Training time was unpaid. Originally, raters could work as many hours as they wanted, subject to work availability. Around the end of 2013, they added a 40 hour/week limit because they were afraid a court or regulator might rule the "contractors" were employees and force them to pay retroactive overtime.

If a state dared to go after Leapforce or Google for misclassification, they would leave that state and fire the workers there. No severance, of course. There were numerous states they did not hire in, including their home state of California.

Task availability was usually very good for US English raters, but there were definitely periods (often around holidays) where that was not the case. There was never any communication about projected task availability. We certainly would have appreciated a quick heads-up of "hey, many of our engineers will be on vacation between mid-December and mid-January, so expect task availability to be light. Tasks will be added daily at 8AM PST, but probably won't last more than two or three hours."

I understand other locales were not as fortunate as we were for task availability.

Work availability for a particular rater was not necessarily as consistent. If you scored poorly on a monthly quality review or random quality audit, you could be abruptly suspended, fired, or limited to an hour or two a day until the next monthly review. There would be no prior warning of this. There were automated tripwires (which we called the "bot") that would flag and suspend you if your ratings were considered outliers, or if your task comments were too similar (even when there was a batch of tasks that were very similar), or if you worked too fast, or if you worked too slowly, or a million other reasons. If you were flagged by the bot, you had no work until the Leapforce quality people got around to reviewing you. There was no compensation for missed hours even if it turned out you hadn't done anything wrong.

Even more insultingly, Google loved to use quality reviews to informally implement guideline changes without actually changing the guidelines. If you weren't good at guessing their future direction, you could easily have your hours cut for the next month.

Around 2017, Google realized they (not just Leapforce or Lionbridge) could be sued for this misclassification nonsense, so they forced Leapforce and Lionbridge to convert their misclassified contractors to employees. Leapforce morphed into RaterLabs in what appeared to be some kind of liability-dodging scam. As employees, they cut us to 26 hours/week so they wouldn't have to pay for health insurance. There were still no benefits other than the minimum required by state law, if you were lucky enough to be in a state that required any. Nominal pay rates were cut in what seemed like a random and inconsistent (not even location based) way; depending on the cut, you got a few cents raise after accounting for the self-employment tax you no longer had to pay (which RaterLabs was quick to point out), or a pay cut.

There was a rater revolt around that time, but nothing seemed to come of it.

Appen bought Leapforce and RaterLabs later in 2017. Leapforce founders Daren Jackson and Caine Lai made off with a pile of cash. His employees weren't so fortunate. Now people are making $10/hr. for a job that paid $13.50/hr. in 2009, which would be the equivalent of $18.38/hr. in today's dollars.

It doesn't work that way. Contrariwise.