Case Study: Accuracy & Fraud on MTurk

Interesting story -- I've attempted to use mturk a few times in the past, but it is actually a lot of work to get it right. Good instructions are really important, as well as a way to measure performance.

Good luck to them as I think they might have made mturk easier to use for people like myself who see the value but couldn't figure out how to "make it work."

I just used Humanoid for the first time. I had a job that would have taken about 5-10 hours (look up 50 ad agencies, record their Twitter account, blog URL, and their CEO's Twitter account and blog URL).

Humanoid accomplished the job for under $35. Considering I would have had to do it myself, or nag an intern for a week to do it, I consider it money well spent.

P.S. I don't know what results I would have gotten from going to Mturk directly, but the results from Humanoid were better/cheaper than my old habit of outsourcing to an Indian body shop.

(Disclosure: I am an investor. But I'm giving honest feedback.)

Great data, very cool how sophisticated algorithms can increase performance dramatically for human-powere tasks

What does the author mean by fraud? Do people submit junk results and hope they slip through the screening process, or it it something more insidious?

"Humanoid’s software identified 8% of the workforce as engaged in fraud." <--that is a big number!

Fails to mention how much they cost on top of the mturk prices to the client