A Plea to Amazon: Fix Mechanical Turk
behind-the-enemy-lines.blogspot.com
behind-the-enemy-lines.blogspot.com
I spent $300 in one day on Turk and Amazon gave me a personal account manager... And that's the problem, if you spend $300 on Turk in a day you're a high roller, Turk just isn't that big of a market. I've had days when I've been responsible for 20% of the HITs in the system and I'm just a poor computer programmer who writes code for a penny a line. I just got an email the other day from a Turk who was begging for more HITs.
Personally I've got no problem with the API and for certain kinds of tasks I've got good answers for the quality problems. For other ones, I haven't figured it out yet.
If you think Amazon's offering a service that's too bare bones, you ought to start a start-up that layers services on top of it. It may be a tough row to hoe though... My layer that interfaces to turk is specialized to my purposes and isn't spiffy, commercial and easy to use. It would probably cost you 20-50 times as much to develop something "usable" than it cost me to develop something that fits my workflow. Unless you can amortize this cost over a LARGE number of customers, your spiffy new system is going to cost more than it costs to pay the Turks. And good luck getting Angel or VC money to develop something that might never get all that big... And that will probably get regulated with the government when they find out that you're getting people to work for you for $2.50 an hour!
For now I'm tickled pink because I can get labor cheaper than my competitors can -- I can escape the economic problems that keep silicon valley companies going in circles chasing each other's tails.
What I really wish is that I could pay people Facebook credits to do work for me: my guess is that I could be paying people $1.25 an hour that way... However, Facebook knows the first thing people would do if it was easy to pay credits out to people is create gambling apps, so they don't make it easy.
It's not that I'm angry at Amazon or anything, I just wish that they would come to the conclusion that our money is just as green as everyone elses.
I see no reason why they can't open it up to Canada. Terribly frustrating. Take my money!
In particular:
The CSV parser doesn't work / is too strict / something. The only way to upload complicated data files in as "JSON". The only problem is they don't accept standard JSON. You cannot have a comma separating the rows. (!?) I have no idea how much time I spent trying to figure this out.
Additionally, if your task is too hard, and more than 50% of judgments are rejected, they auto-pause your task. When you resume it, it will simply pause a few minutes later. So, for certain tasks, you simply cannot finish the job. Their rationale is that CrowdFlower pays for the judgments that are discarded, and you pay for only the judgments they keep, so if they discard too many judgments they lose money. I would rather just pay extra money for tasks that are very hard.
We find it restricts the kinds of work we like to do too much - but it also restricts it to the kinds of work that Crowdflower can reasonably return good results on. So that seems like a good trade off.
I recommend Crowdflower as a starting place to anyone thinking of using MTurk.
I like Panos have been doing this for more than 4 years, and I've seen no silver bullets. I'd like to think we're (CastingWords) doing it better than anyone else, and I don't even know what a silver bullet would look like at this point. Though I have to say that Panos's list is a good start.
https://spreadsheets.google.com/viewform?formkey=dDNkei1Hb0l...
Using the RTurk gem and my recently released Turkee gem ( http://github.com/aantix/turkee ), I've been able to quickly throw together a demo Rails application that allows Turkers to vote on previous Turker submissions or use other turker's submissions as the basis of their own own submission.
With my experience under this type of interface most Turkers will vote for one of the previous submissions but if you offer a small bonus for the highest voted submission, more turkers will take a chance at submitting something original (even if it's a trivial 50 cents bonus).
The demo's code is available under my github account (aantix/Turkee-Iterator). Here's some of screenshots of my demo; the top image shows my initial parameters for setting up my Turkee task. The second screen shows the responses I received from the Turkers. The third screen shows the form that was presented to the Turkers (either vote or submit a joke).
Right now I am writing a game that generates content using a couple iterative approaches to content generation with Mechanical Turk (hence my interest in this area).
Don't hesitate to drop me a line if you have any problems with getting my demo running.
If you're looking for a solution to some of the problems addressed in this article, his API (and the corresponding Rails gem) are a good place to start.
An interesting finding is that it costs about $5K-10K to open a US bank account from overseas, post 9/11. Pre 9/11 it was probably around $100.
Although you can use MT credit to buy stuff out of the Amazon store.
As far as protecting workers, in a survey I ran last year, there wasn't any evidence that workers on MTurk feel more exploited by their online bosses compared to their offline bosses:
http://www.technologyreview.com/blog/arxiv/24646/
Though obviously those that feel particularly mistreated could be selecting out of the market.
You will see that the low wages are a direct result of a lack of a good reputation system