Hacker News Reader

Comment by tansey | Hacker News Reader

This article is an excellent summary of all the frustrations I've encountered using both oDesk and Mechanical Turk.

I'll add one more issue with MTurk: binary decisions. Some of my tasks can't easily be broken down into accept/reject, or sometimes a worker gets something close but not all the way there. I want to be able to say "hey, you did pretty well, but please make these small changes and we'll accept it."

But that's not possible right now. You get the HIT and it's accepted or rejected. The result is that we have to sometimes reject workers who are making an honest effort but didn't sufficiently complete the task or we have to accept sub-par answers so as to not have to deal with worker blowback.

View on HN

Let me suggest one solution for such cases: Use iterative tasks, in the spirit introduced by TurkIt.

For example, you want to create a caption for an image. You let a user create a caption. Then you take this caption and give it to another user, asking the user to improve it. Take the two versions and ask other workers, "which of the two versions is better?". Iterate until no improvement is possible.

Not a trivial setup, but gets around the binary accept/reject decisions problem and generates results of significantly superior quality.

Good idea. I've also thought about this for data entry/analysis type tasks. It is fully expected that a worker on any level won't be perfect, especially on these types of services. An approach I've considered is to have have 3 workers do the same task then compare results and take the majority answer which should be close to if not 100% accurate. Overtime you could even rate workers for accuracy and recruit/pay more if possible.

On a very selfish note, take a look at these:

http://qmturk.appspot.com/ http://code.google.com/p/get-another-label/

You may find the code useful for what you are trying to do.

In addition to the iterative approach, you can also offer a base rate and accept all the tasks, even if they're not quite perfect, then pay a bonus to the workers who really nailed it.

That way you're not rejecting the workers who are making an honest effort, but the better performers are being paid accordingly.

Reply on news.ycombinator.com