I hired 50 people today. This is what they look like | Hacker News Reader

I hired 50 people today. This is what they look like

thestever.net

166 pointssteveeq1118 comments

Andy Baio did this last year with the additional task of including "why do you turk?"

http://waxy.org/2008/11/the_faces_of_mechanical_turk/

Interesting to see how the demographic has shifted in that time period.

Regarding the bit about crowd-sourcing ideas; Andrew Hyde used Turk to solicit startup-ideas: http://andrewhy.de/using-mechanical-turk-for-ideas/

Or time of the day?

Andy Baio's project took place over the course of 3 days; it's not explicitly stated how long this task took to complete though given how high the pay was (relatively) I could see this job filling up in an hour.

(Edit: or 3 hours - pay seems to be a large factor, I wonder if by paying less one gets more people who do the task out of boredom rather than necessity and how that correlates to quality of results.)

It could have something to do with the time the job was created. As the author mentioned in a comment, after 3 hours the job kinda died. So If he posted it in peak hours for India, than it's not surprising that most of the respondents were Indian, while the fellow from your link may have posted it in peek hours for a more "white" timezone.

What we really need I supposed is one of these every hour for 24 hours. Then we'll get a better idea of the demographics of MTurk.

This is similar to a short survey I ran the other day: I wanted some stats about how many people used wedding gift registries, so I put an ad on Facebook (targeted at married females in NZ) that said "Answer 4 questions about your wedding and be in to win $50".

70% of people who clicked did the survey, which is a really high "conversion" rate -- I suspect the $50 prize draw really helps. I spent about $0.25 per click, so all up 200 survey results cost me about $120.

So did you actually give the $50 prize to anyone?

Yes -- though not yet, because they (weirdly) haven't replied to my "you've won the $50 survey prize" email. Either they thought it was spam, or they typed in their email address wrong. I'm guessing the later, and will move onto the next randomly-selected participant in a couple of days.

http://xkcd.com/570/

Facebook used to have a really useful (and inexpensive) Polls feature that allowed you to get something like 1000 or 5000 responses from your demographic to a multiple choice question. They pulled it for some reason though. http://www.washingtonpost.com/wp-dyn/content/article/2009/01...

I'm surprised that this made it to the top of HN. I did something similar for my friend's birthday last year, but I never considered it particularly newsworthy. Another fun project was proving to our product manager that he was completely wrong (about listing overdue timesheet dates in reverse chronological order) by submitting the question to Mechanical Turk and getting instant user feedback.

Does the popularity of this link mean that there is enough interest in Mechanical Turk to justify the creation of web apps around it?

The HN front page has been defying all logic as of late...

Maybe part of the MT task was to register on HN and up-vote the post?

No, I didn't do that. At most I put a reddit and a digg link on the page.

http://www.dilbert.com/fast/2007-09-03/

Is it common for Turkers to repeat tasks? These people used the same sign: http://www.thestever.net/viewer.php?num=1 and http://www.thestever.net/viewer.php?num=9

Yeah, it is. There were a lot of bad submissions. I'm just experimenting with Mechanical Turk right now. Initial impressions. . . 1) a lot of fraud (about half sent in pictures of their cat or something) 2) most people are from india 3) Most of your replies come in the first 3 hours or so. After that it wains. You have to resubmit after this 4) There were repeat submissions. Three of them looked like they are from the same room, same camera. But people claimed different locations.

After that it wains

wanes

A wain is a kind of wagon or chariot.

Can someone please tell me why this submission is interesting to HN? (Obviously it is, I just can't figure out why.)

Mechnical Turk is kind of interesting. It's one of those things that I'd really like try out, but have no immediate use for. I'm sure I'm not the only one. (In fact, most of the cool projects I've seen using MTurk are of the Gee Whiz variety)

Also, people like to look at pictures.

Try soliciting suggestions for birthday or holiday presents.

Or, try soliciting feedback from 100 people on the clarity/ease of your company's sign-up process, introductory video, etc.

Many of the comments will be worthless, but you'd be surprised at the quality you get from some people, even at rates as low as $0.05-0.10.

Mechanical Turk is a fascinating tool, maybe this is not the most interesting use, but it still make hacker drool. When you can't have AI, you use AAI (Artificial AI).

You can use it to train pattern recognition. It sure beats doing the work myself.

It helps to be very specific in what your HIT is and reject submissions that don't complete the job.

I think if you are ambiguous in your request then it is unfair to reject submissions that follow the letter of your request but not the intent.

That's making other people pay with their time for your own lack of specificness.

The following guy made me smile. =]

http://www.thestever.net/viewer.php?num=27

My favorite as well. Reminds me of an evil genius in-training from Captain Horrible's Sing Along Blog or something.

This makes me curious about Amazon's vetting process for requests.... taken one step further, I can imagine all sorts of ridiculous things to ask people to do.

Why would Amazon vet the requests? People can choose whether to take part in any particular project.

What if somebody asked people to write something profane? Or political? Or perhaps deface the image of an icon or religious leader? Once you move past written messages, you open up all sorts of avenues. Crowd-sourced politics? Crowd-sourced porn? Crowd-sourced violence? Etc. At what point does Amazon need to filter these requests?

Perhaps Amazon could crowd source this?

After all, for someone to get anywhere doing ill with MTurk the request must be seen by people.

When it's illegal? There's a huge difference between crowd sourcing violence and writing something political (which a lot of people already get payed to do), ones organized crime the other is not.

I don't see any problem as long as they're not asking them to break any laws. Ultimately it's up to the parties involved to obey the law though. If someone puts an ad for a hitman in the help wanted section of the newspaper, should the newspaper be held responsible?

On demand viral marketing or protests...

I do think it is very interesting to see the faces behind Mechanical Turk.

> Crowd-sourced politics? Crowd-sourced porn?

So?

While I don't doubt that everyone has their price... I think you would find it cost-prohibitive to pay enough people to do something like that to have any sort of real-world impact.

People are not going to go out of their way, or risk jail time, for $1.00/task. You'd need to pony up a lot more money (like, I'm guessing, 100's of thousands of dollars) to get enough people to violate basic principles on a large enough scale for this to create some sort of shift.

If they're in India, and breaking laws in say the US, I doubt they'd care too much.

The one fascinating angle I could think of is crowdsourcing something that is harmless and legal if done once in isolation but becomes harmful and dangerous when done on a huge scale. My brain is totally failing me here on a coming up with a good example but it seems conceivable (and diabolical!)

One papercut is a nuisance 100 million is another thing altogether.

Something with stock manipulation could work. Buy a ton of options in some company, like Palm for instance. Then on the release of some hot new gadget, like the Pre, pay a ton of people to wait outside the doors of the vendors. Initial buzz will be that Palm is going to strike it big. Cash in.

However, there would be a record of the entire thing in Amazon's system, not to mention that a huge audience of potential Mechanical Turk workers would see the offer - it seems likely that at least one would be suspicious and try to notify an authority.

Is that actually illegal?

Yes.

What if they told people to stand in line at the store next to Palm? Since there's a finite amount of space, they'll naturally overflow. To an outside observer, it looks like all these folks are standing in line for Palm - particularly if you do the same thing at 1000 locations nationwide, and let journalists connect the dots.

It's also far less traceable. Instead of paying 50,000 people to stand in line outside of Palm, you're paying 50 people each to stand in line outside of 1000 different local businesses, whose only connection is that they happen to be next to a Palm store. A just-as-plausible explanation is that you're organizing a "buy local!" protest that day.

You could also pay people on Amazon Mechanical Turk to pay turkers to stand in line, making things even harder to track. Now instead of one guy paying 50 people each to stand in front of 1000 different locations, you have 50 guys, each one of which is paying people to stand outside Subway, or McDonalds, or Verizon, or the corner bookstore, or whatever happens to be next to that Palm store.

This reminds me of a guy which used Craigslist to hire decoys for armored truck robbery getaway:

http://news.cnet.com/bank-robber-hires-decoys-on-craigslist-...

If it were in a movie you'd say 'no way', but to see it in real life... I'd hate to be that guard, being pepper-sprayed is not nice, but otherwise this 'heist' has me with ambiguous feelings towards the robber.

Pay people for a DDOS?

That's what botnets are for.

so you won't have to pay the people?

My understanding is you can hire botnets. It would be far cheaper than Mechanical Turk for something like a DDOS.

One obvious application is using MTurk to solve CAPTCHAs.

Isn't that what 4chan is for?

How about you pay everyone in China to jump up and down until we crash into the moon?

http://qntm.org/?moving - last entry at the bottom.

That naively assumes that you jump up then fall. They could simply climb up steps and drop down, thus resulting in a net movement. Over time we would move into moon collision orbit.

Uh,

Perhaps considerations of aiding and abetting prostitution, kidnapping and drug-dealing would enter if the job requests were entirely unvetted?

Uh, and I think India has labor laws too.

Surely they should be vetting to ensure the job is legal and doesn't break any laws.

If they were to do so, beyond merely making people agree to terms of service that prohibit such things, they would very likely open themselves up to a lot more legal liability. If they did claim to review and filter projects, and they missed one, they could then be liable for whatever horrible things happened as a result of someone performing the task. I'm pretty sure being a neutral exchange, where both buyer and seller beware (though each has processes for remedying lack of performance or lack of payment), is a safer model for Amazon.

On top of the seeming additional liability Amazon would be inflicting on themselves, I can't see any upside to Amazon going to extra lengths. It would anger project creators to have to wait for human approval for every project, it would likely reduce the number of jobs underway at any given time (making it less profitable for them and less lucrative a market for service providers), and it would increase the expense of running the market at human scale rather than computer scale (even if they somehow figured out how to use Mechanical Turk itself to provide the vetting). In short, vetting jobs would be a recipe for a fiery death for Mechanical Turk.

There is a request up now looking for someone to take a picture of themselves dressed as a terrorist. Many others ask people to fill out a quick survey or enter in an email for lead generation/reward websites.

I think you failed in your objective to say 'hi' to that actress, instead you have made lots of other people say 'hi'.

So who won the $50 bonus?

http://www.thestever.net/viewer.php?num=4

I am writing her now.

She is my sister. Pervert... :D

First thought: seems like Mechanical Turk is popular in India.

There also appear to be MTurk "shops"--some of the people are clearly sitting at the same computer (some even used the exact same sign...)

Yeah I noticed that too. There's also some location inconsistencies!

http://www.thestever.net/viewer.php?num=33

http://www.thestever.net/viewer.php?num=34

http://www.thestever.net/viewer.php?num=35

Here as well, look at the closet on the right and the pictures on the wall.

http://www.thestever.net/viewer.php?num=10 http://www.thestever.net/viewer.php?num=14 http://www.thestever.net/viewer.php?num=45 http://www.thestever.net/viewer.php?num=42 http://www.thestever.net/viewer.php?num=41 http://www.thestever.net/viewer.php?num=32 http://www.thestever.net/viewer.php?num=31 http://www.thestever.net/viewer.php?num=30 http://www.thestever.net/viewer.php?num=29

That doesn't indicate a formal "shop" to me.

If you're getting $1 for each picture of a person holding a sign you submit, you might as well have everyone else in the room take a turn.

That makes sense. $1 is 40 Indian Rupees, and $50 (the prize money) is 2339 Indian Rupees at current rates. Neither amount is an insignificant amount of money in India.

If $1 = Rs. 40 Then $50 = Rs. 2000

Yes. Neither amount is insignificant. But I still wonder, why only India? People from Pakistan, China and even Sri Lanka , Singapore are in +/- 3 hrs. timezones.

The timing of the post would be crucial to understand this.

Indians speak English.

Fewer than 10% (almost all of which speak it as a second language)

That's still about 100 million people.

What's that expressed as a percentage of Indians with Internet connections at home?

So do people from Singapore, Pakistan, Sri Lanka, Bangladesh, Hong Kong, Thailand, Malaysia, Iran.

FYI, $1 is insignificant in Singapore. Average household monthly income probably around 4000 USD (personal estimate, I live there, didn't look for official data)

Agreed. I had mentioned Rs. amount there - as a reference to the parent post. quoting Indian Rs.

I wanted to highlight - the time as a factor for other countries. Timezone.

What irks me that as an Indian I can't create a job on MTurk! I would love to try MTurk but it is only open for US citizens..

It shouldn't just irk you as an Indian, it irks me as a dutch person as well.

If a service finds people from a certain country good enough to be 'minions' how come it finds them not good enough to be 'bosses' ?

This is actually a really nice opportunity to give amazon a run for their money and to set up a competing service that operates world-wide on both the give and the take side.

There are employment laws in the UK which mean a Mechanical Turk-like system wouldn't be able to have jobs listed for such low amounts - I think it comes under the minimum wage laws.

Because of this and similar laws in other countries, Amazon have kept job-posting MT accounts limited to the USA at the moment, which doesn't have the restriction.

I don't know enough about Dutch or Indian law to know whether a similar system could be set up within them, but there are problems with rolling something like this out globally.

Minimum wage laws only apply to a situation where there is an employer-employee relationship between the two parties, this is much more like a free-lance situation.

Aaron Koblin, of the flight patterns visualization fame (http://www.aaronkoblin.com/work/flightpatterns/index.html) has a few art projects based on MTurk. Bicycle Built for Two Thousand was a project where each worker was asked to sing a tiny part of a song, without knowing what the song was, then the complete song was reconstructed: http://www.bicyclebuiltfortwothousand.com/

funny, I've been running a HIT recently which asks turkers to wish my friend a happy birthday. http://happyturkday.tumblr.com/

This is not nice: "I'm offering a $50 reward to the guy with the best photo, so have fun and be creative" They thought they had a chance of winning.

Blows my mind how 4 out of 50 (8%) did not even complete the stated requirements.

Also crazy how all many of these people are Indian.

4 out of 50 is 8% not 12%.

I think he paid them all $1 for the task, with the added bonus of possibly getting $50 if they win. Why is that not nice? They got paid, and entered a contest. $1 for a HIT on Mechanical Turk is pretty good, especially considering the amount of work they needed to do here (which is not much).

are you kidding me? it is a fricking $1. Even in a developing country is not much, especially with the dollar going down.

Don't be surprised when people don't take your requirement that seriously.

I've also seen people making 100k+ who can't follow simple instructions or be bothered to read requirements.

45RS to take a photo and post it online is a good deal. That buys you lunch (or food for your family).

1$ is the salary for one hour work for a maid in Shanghai... And Shanghai is not exactly the cheapest part of China.

$1 seems to be pretty great for this task. I just read some HIT descriptions, and there were A WHOLE LOT of "write 200 words for $0.03" jobs. Which really sucks if you ask me.

The funny thing is, every single one of those participants thought it was worth their time. They looked for something to earn money, and found this. They clearly thought the time/money tradeoff was sufficient.

If you can earn a buck doing what a lot of (generally wealthier, admittedly) people do for free, why not?

Over 50% did not do the required task. A lot of people sent me a picture of their cat or something.

You could run another HIT to get people to verify the submissions and reduce the error rate further.

Cool experiment. Thinking of using Turk for photo approvals on my site, but the accuracy issue leaves me slightly hesitant. Has anyone had experience doing this?

From what I read the standard technique seems to be let two people review everything. When there's a discrepancy you can add more manpower (and take the majority view or so) or look at it yourself.

Yup. There are a couple of language translation services that use techniques like that.

A problem is that there are rooms of people turking together. They'd probably all submit the same translation, for example.

I guess you should flag exactly similar translations. Though I do not see how you can do this without relying on security by obscurity.

What I did was to have up to five person work on a hit. Many simple hits are done at 0.05$ so it's still not too expansive.

For text submissions, you can enforce validation rules (simple and regex). So you don't have to manually reject non-conforming work. You can also have the Turker fill in their work on a webpage that you control, so validation can be much more advanced. I never went all the way but sometimes I think about it.

Hey ashishk, sorry for using HN to contact you, but I can't seem to find your contact info anywhere.

Our company provides very affordable and accurate photo approval services. Photos are manually approved and we can handle anywhere from 10-1,000,000 photos per day. If you're interested, feel free to email me (email address is in my profile).

funny how http://www.thestever.net/viewer.php?num=33 and http://www.thestever.net/viewer.php?num=34 have the same color background (even the dot on the wall by the upper right hand corner is the same) but the signs they are holding are across the world from each other...

Would you do that for a dollar? I might if I knew where I had a spare piece of paper and a Sharpie, but I don't.

The payment for most of the tasks on Mechanical Turk and other things like it is just not worth the time for almost any task I've seen. Which is really sad, because it'd be awesome if it were worth it.

The payment for most of the tasks on Mechanical Turk and other things like it is just not worth the time for almost any task I've seen.

While that may be true for you, there are plenty of places across the globe where $1 (with a chance of $50) is not an insignificant amount.

I would if I just happened to come across the posting and just happened to have paper and a sharpie and a digital camera next to me.

It's interesting to read all the comments that say "$1?!?! You're ripping off these people! Why would anyone ever do anything on Mechanical Turk?" How many hours do y'all spend on Hacker News? How long did it take you to write those comments? How much were you paid for it?

In strictly economic terms, Turkers are getting ripped off a lot less than news.YCers are. After all, $1 is infinitely more than $0. We labor for free so that Paul Graham can have a viable online community and pick of the top posters for his seed capital firm.

This isn't really labor, first of all. For most of us it's both a leisure and educational activity that we participate in not for PG's benefit, but for our own benefit by learning and socialization. Making a sign that says "Hi Lili" and taking a picture of it is much less fruitful, even if it nets me $1 in money. The things I learn on HN net me much more money in the long run.

But we have no way of knowing if Mechanical Turk participants think of their turking the same way...

Except perhaps the ad-hoc (non-rigorous) studies that have been done before asking people why they turk. Someone mentioned last years submission of such a study further up the thread.

Of course you may be being philosophical about it. In which case, no we have no real way of knowing if they even think.

The taking a picture task to could be though of as fun and something different compared to a lot of the jobs on there. So in this case someones motivation might not having anything to do with the dollar on offer.

$1 - $0 = $1

$1 / $0 = undefined ;-)

And hey - if that $1 takes you 5 minutes, then that's $12/hour.

I'm pretty sure that might be above the US minimum wage.

The economy is saved!

Oh, maybe not...

I hope I'm not sounding offensive. But what is the author trying to prove here? It's a well known fact that majority of the people on mechanical turk come from developing / under developed countries.

I didn't think there was any commentary on where the people are from. Did I miss something?

I didn't mean to be on the offensive side or to start an argument. I was just asking a question. Refer the author's reply. http://news.ycombinator.com/item?id=1010388

No one has accused you of being on the offensive side, I don't think, and I wasn't arguing with you. I just didn't see any indication that this project had much to do with the location of participants. So, I just wasn't sure what you were getting at with your "But what is the author trying to prove here?" query, since it seemed to me that the guy was just trying out Mechanical Turk and documenting the results, without any social or political commentary that I could perceive. It doesn't seem like he was trying to prove anything.

Does the author say anything at all about race or country? Besides most of them seem to be from USA.

They are lying, however:

http://www.thestever.net/viewer.php?num=33 http://www.thestever.net/viewer.php?num=34 http://www.thestever.net/viewer.php?num=35

Good catch.

What he proved is that with $100 and some imagination he could learn a lot about Mechanical Turk.

As a bonus, he got some nice smiles out of the process ^_^.

Reply on news.ycombinator.com