Poll: What experiment to run on AWS mechanical turk? | Hacker News Reader

Poll: What experiment to run on AWS mechanical turk?

11 pointsadatta0213 comments

After our experience using the mechanical turk we've become a bit obsessed with trying to find a really awesome use case for the service - even if its just experimental.

We have some downtime right now and we are narrowing down the experiment between the following:

1. Some sort of cooperative art project - inspired by http://www.tenthousandcents.com 2. A massive chess game that forks every time a move is made (probably a poor idea) 3. Some implementation of the Prisoner's dilemma game. 4. Any time of game to get a sense of how risk seeking or risk averse the Turks are - possibly a tie in with the Monty Hall Problem.

Most of these ideas are centered around game theory but we are pretty much open to anything (within reason). Any votes or opinions on what would be the most interesting/entertaining?

-Thanks

View on HN

I'd love to see a prisoner's dilemma experiment via MTurk. These tend to be financial in nature, though; can you set variable payment rates per task, depending on external factors?

you can pay out arbitrary bonuses once they've answered.

so you could easily do single-round prisoner's dilemma, and have amazon pair up 2 (different) folks at a time, and when you get the results back, look to see what the folks in each pair did, and award bonuses appropriately.

i'm not sure if there is a way to do an iterated version.

A friend last week told me about a business he's working with. They do Natural Language Processing to scan thousands of articles about a specific topic/product to see if they are positive, negative or neutral. They are then hoping to turn around and sell the business intelligence.

I was wondering how it would compare on a cost/effectiveness basis with the Turk.

Let us know what your experiment is and how you fared.

See if you can use MT to provide the fitness function for some genetic algorithm, evaluate the results, tune it, and profit!

Some months ago I tried to set up a project to transcribe Alexander Shulgin's Lab books but Mturk wasn't available outside US.

http://www.erowid.org/library/books_online/shulgin_labbooks/

Go meta - use Mechanical Turk to ask what would be a really cool idea to run using Mechanical Turk.

Liar's poker. Ask 2 people to pick a number and the one that picks the lower doesn't get paid. The number of people that actually play the game is the result of the experiment.

How about a dollar auction?

How about creating an experiment and designing it right on turk? You create a rough sketch and then you auto-correct it using people?

You mean keep tinkering with the study until it "works"?

If you're not careful, you could get http://en.wikipedia.org/wiki/Publication_bias on a massive new scale...

It would be interesting to try to find scientifically optimal ways of doing study iteration, or approaches to interpreting the complete (experiment_rev, results)-list from another researcher...

I'd pay $.05 to get the them to up vote my site for certain search terms on Google's new "Search Wiki". See what happens.

What was your previous experience?

we used the Turk service to basically do OCR on about 3k images - http://shout.setfive.com/2008/11/20/artificial-artificial-in...

Reply on news.ycombinator.com