How Amazon's Mechanical Turkers Got Squeezed Inside the Machine | Hacker News Reader

How Amazon's Mechanical Turkers Got Squeezed Inside the Machine

spectrum.ieee.org

78 pointsheadalgorithm32 comments

I tried to do some Turking just for a little side-cash.

I stopped after I realized that buying a higher tier of tax software just to handle multiple income sources ate at least the first $15 of income (which is a significant time investment in Mechanical Turk).

In other words the income was so low that even paying the taxes on it ruined the whole venture. Even Bing's rewards have a higher rate of return. The $2/hour from this article entirely mirrors my experience (although my average might be a little lower).

I have an anecdotal story on the other side. I had a small startup and we had a list of URLs. We loaded them into mTurk and asked turkers to visit site and find an email address (probably could have automated this ourselves, but we didn't). Most of the responses were just info@<<websitename>>.com and we could see they completed the task in <10 seconds. So although info@ is a legit answer, after checking a few we found out the turker just guessed it would be a suitable answer and get paid for it. We basically couldn't trust any of the answers and just walked away from that service.

Amusing. Couldn't someone on the other end automate this, then cash in on it? I wonder if there's some potential in this thought: paying for automation as a service, based on the volume of output...

So... Fiverr or Upwork?

Those are paid on a per job basis. Different incentive structure.

Here you would basically be defining how much you'd be willing to pay per work item, regardless of how it's done; you wouldn't get the secret sauce, just the black box output.

It's not really a great idea, but in fairness I don't think it's much worse than what's out there either.

There are quite a few "scraping" companies out there that do this. They generally focus on getting you the data in an automated fashion, but it is black box and could be manual if they wanted.

I sense an opportunity to arbitrage mTurk -> Automation -> mTurk. Repeat as many times as necessary for a successful exit (IPO vs. Acquisition doesn't matter, either are acceptable)

What's even worse is the people who do that simple guessing vs. actually doing the task ruins it for others -- eventually the Amazon mTurk AI thinks that your correct answers are 'wrong' due to a mass of others garbage responses. Thus your accuracy rate goes down and you can't qualify for many or most of the new mTurk tasks.

You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to verify the response quality. Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.

You can also use qualification tests (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanical...) to require Turkers to demonstrate competence before doing tasks. I'd imagine you could inject a certain percentage of known-answer tasks into the queue as well, as ongoing quality control.

> Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.

I wonder if there's been research into the effort in time or persons to perpetuate this fraud. Knowing Amazon, I'm certain they've analyzed this internally if not constantly at least a few times over the years. I wonder if there's a public attempt at the same analysis.

Seems to me the answer to this is a sliding scale of compensation that rises for consistently valid responses and decays to zero for these sorts. Of course, this can be gamed as well, but it seems like everything's a Red Queen's Race these days so the only way to avoid^H^H^H^H^Hminimize one's losses is to race.

Did you mean "... is to [not] race"?

No, I meant exactly what I wrote. For example, if you were to try to duplicate Google today, you would drown in content farms and other sorts of BlackHat SEO because Google has been engaged in just such a Red Queen's Race with the BlackHats for over a decade.

So even if you come up with a scheme that rewards WhiteHat Turks, the BlackHats will inevitably crack it unless you insure it's a moving target.

Have seen similar things, as it sounds like a great way to use Mturk on the surface. Perhaps the answer is more direction and qualifiers, such as "must be a name, cannot be info@ or editor@"

Side note, if anyone has a way to automate the above in a way better than Hunter.io, there's big money in that.

Had we known about Hunter.io we would have used it. We had a small list of URL (~500). Would have been the small price to simply have it automated by a specific program rather than humans.

I tried MTurk as well out of curiosity. I only made a few dollars before giving up, but my observation was that like anything else in life, you get what you pay for. High paying jobs on MTurk require a impeccable record of a high number of accepted jobs and a near perfect acceptance ratio. You slowly (I can't emphasize how slow this is) build up your reputation as you complete jobs for pennies until you get to the category where you can accept jobs that pay $100 dollars or more.

Turkers who work hard to get the reputation to do high paying jobs won't risk losing the reputation by cheating. So if you want good results you need to pay up to get those Turkers on your job.

Seems like there should be other ways to boost your reputation. Maybe putting a deposit down which you can lose if your acceptance ratio gets too bad.

Wtf is with people suggesting contractors getting paid less than minimum wage should be required to put deposits that can be stolen from them if they don't work?

It's this just schilling or...?

No it's the standard solution for quickly establishing some reputation, prove that you have something to lose.

Wouldn't tax software make for a perfect open source project? Has this been attempted? Any money I give to tax preparation companies upsets me, and my taxes are as simple as it gets.

Tax software would make for a terrible open source project because of the intersection of domain expertise: to do it right you need CPAs and tax lawyers. To do that right, you have to pay them (because there's no "open source" equivalent of a trained and certified CPA or lawyer). If you are paying domain experts, why shouldn't you also be paying software developers? Building a business plan for something like tax software is already hard enough, especially with such entrenched players already, but building a business plan for something like tax software that is also open source is extreme hard mode.

So many social science papers rely on MTurk which is concerning given the poor compensation. With such low payment structures, MTurk is incentivizing people to rush through responses and give responses that they anticipate the researchers want to see.

(Note: I am an academic social scientist who has done an MTurk pilot but never published anything based on MTurk responses)

Yes, this is true, but this is true outside MTurk as well. If you rely on the YouGov panel (people complete surveys to get points for t-shirts), or really anything else, there's a strong incentive to cheat.

This is why good survey research is going to involve attempting to bound fraud through a variety of measures: attention checks ("What's your favorite color? Ignore this question and answer yellow."), looking for straight-lining (respondents always picking the leftmost answer), looking for unusual contradictions (i.e. asking the same question in two, opposite ways and looking for people who don't have the expected relation in their responses), looking at the distribution of completion speed and scrutinizing the lower quantiles, attempting to log participants who take the survey multiple times through dummy accounts, etc.

Oddly, my experience was that Turkers were fairly conscientious. I know one theory for this is that many Turkers are in fact Turking on the job, and so their reserve wages are not about what they are being paid for the MTurk hit, they're about what they're being paid to sit in their desk and not work.

> "What's your favorite color? Ignore this question and answer yellow."

I’ve noticed something similar to that in some of the YouGov surveys I’ve answered over the last decade. I wonder what it says about me that I sometimes write, in the errors/comments section at the end of the surveys, that I have seen surprising questions such as “what do you think about Channel 4?” when I have previously answered that I haven’t watched Channel 4 recently.

In this scenario, one might not watch Channel 4 for a reason, so asking for that reason is sensible. “User A doesn’t watch channel 4, this is what they think about it.”

Could be; while it seemed very wrong when I saw it, I didn’t record the exact words and it may have made more sense than it seemed to.

Just last week, I saw a presentation about some research results that used MTurk subjects. Some of the results were counter-intuitive, albeit fairly consistent across experiments. However, the subject matter was such (basically trust in the results of ML models with differing degrees of complexity/transparency) that I couldn't help wondering if the results might be greatly affected if the outcomes were genuinely important to people.

As you say though, it's the same problem with just about any study of this type whether MTurks, some other online panel, or students earning a little bit of beer money.

I used to work in assessments (educational testing, employee satisfaction, etc.), and the MMPI and other psychological testing would sprinkle in items like "I have never had hair on any part of my body (agree/disagree)" and use them to calculate an honesty/reliability scale, to help assess whether the results were usable or the respondent was giving random or misleading answers.

Many of the problems in comments are from new requesters/workers using/having poor qualifications or no qualifications, having no quality control, and not researching the platform before using it. If you think you can just dip your toe into the Mturk pool and profit, you're wrong. There is a huge learning curve for both workers and requesters. If you do not invest the time in the platform, you will get garbage pay and garbage results.

MTurk sucks. I need to have over a thousand video transcripts manually fixed, and so far, only about 15% come back even close. Most MTurkers submit the original, unmodified transcript, or something completely irrelevant. I have to use diffing software just to quickly scan each submission for scammers trying to make a quick buck. Granted, I feel like a scammer myself for paying so little (but the decision for payout is coming from above me).

(I keep reading about it but I never used it)

Does mTurk have some sort of quality-related metrics respectively higher rewards linked to it?

For example when I submit some work can I specify "I want only at least 4-stars-people (out of 5, where "5" is for people that rarely make mistakes) to work on this and yes I will pay 4$ extra fees for 4-stars and 8$ extra fees for 5-stars rated workers"?

Thx

Reply on news.ycombinator.com