Human Workers, Managed by an Algorithm | Hacker News Reader

Human Workers, Managed by an Algorithm

technologyreview.com

80 pointsprayag28 comments

It's all about how much you pay the turkers. I have done many studies on mturk and the quality of results have been very good so far. My secret? Pay people equitable wages.

I always do a small pilot study to find out how long on average does my task take people to do. Then price the HIT so that people can get at least $6 an hour, sometimes much more.

This way, I get high quality results very fast. For example, I posted 100 assignments at 11am today, each took about 10 minutes to do, and now it is 1:40pm, and I have already gotten all the results I need.

Compared with having to pay $20 for people to come to my lab for an hour, mturk is heaven.

That's a terrific strategy, and that's one of the strategies we apply internally here at MobileWorks.

Treating folks fairly is the right way to get good results. We tie price to task time explicitly, to prevent folks from underpaying MobileWorkers. This makes sure results come back right.

You can read our IEEE paper for details: http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-mobilewor...

That is great. When I was working in the Philippines for a microfinance bank I had some of my staff work on mTurk during a quiet period and they were barely earning local minimum wage from the work. Croudsourcing work like this offers great potential to get income into poor, under-employed regions of the world but we can't let it become the new sweatshop.

Full circle.

Remember that the original 'computers' were people who performed computations.

If you recall Feynman's self indulgent book 'Surely you are joking' he has a chapter where he describes how they would write algorithms which were then run by humans operating adding machines while working on atomic bomb related problems.

Is this related to the claim that Feynman was one of the original forefathers of parallel computing?

Hmm, perhaps: there is a passage somewhere about how they set up a scheme where instead of the one person doing all the different computations they set up sequences where each person would only do one computation but for different problems which would circulate through the pool in a staggered fashion of some such.

I think that has to do with feynman's time at Thinking Machines:

http://longnow.org/essays/richard-feynman-connection-machine...

Whilst waiting for my turn to order on a local subs restaurant, I saw something familiar. I was watching Big O live and in action. There were two cashiers. Each one greeted the customer differently. The lady on the right was faster, because she would ask yes or no questions. Then she would ask for payment type up front and prepare the ATM/credit card terminal in advance if the client said ATM/CC. The other cashier would just ask random questions in no particular order, and would wait until the last minute to prepare everything for the payment part of the transaction. She never had the pen at hand. Ever.

Two people. Same training ( I assume ), and two vastly different algorithms operating for the same goal. Imagine if a computer would take in the order. It would closely match the lady on the left POS terminal. She was as close as a computer anyway.

Her routine went a bit like this:

    (start transaction
        (greeting
            (choose sub type
                 (type of bread (white or wheat)
                      (any deviation from the STD ingredients)
                           (? combo meal))
            (? change chips for cookie)
            (? change soda for bottled water)
        (type of payment
             (prepare pen & adjust terminal)
            (? ATM 
               (get card ( swipe ( punch in numbers ( ? accepted || declined (goto start <-- ) give customer card back )))))
     (end transaction)))

I just realized how lispy this came out to be. :)

Interesting that MIT Tech Review wrote an article on this, since a lot of great research (at various universities, including MIT) is now a lot easier due to easily available human task workers. Hopefully this will make research easier, maybe reduce grad-student drudgery, and speed up scientific progress for everyone. (Task work is useful for business, too, but it seems particularly useful in the same ways cloud computing is useful -- the ability to rapidly spin up a short-term task to handle load, without persistent staffing costs.)

>since a lot of great research (at various universities, including MIT) is now a lot easier due to easily available human task workers

This is very true. A lot of researchers today are using MobileWorks and other crowdsourcing platforms. It has a lot of implication in a number of research areas including AI, HCI, Databases, Bio-sciences, Earth-Sciences and a lot more. These are all fields that have used crowdsourcing and have published papers about crowdsourcing.

In fact, MobileWorks came out of UC Berkeley where the founders were doing research in HCI using crowdsourcing before we started building our own platform because we thought the existing competitors were a pain to use.

Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up".

If data quality matters and you can avoid mturk - avoid it.

>Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up".

This is what we are trying to solve. You can get good clean data from crowdsourcing too if you give it out to the most suitable workers and have them work collaboratively.

>If data quality matters and you can avoid mturk - avoid it.

You've identified a great reason to use MobileWorks instead. Crowd work doesn't have to be low-quality; Turk just hasn't done a terrific job of it.

We've achieved substantially higher quality than MTurk by treating workers better, paying fairer wages, and giving workers tools to collaborate.

You get what you pay for. If you pay them pennies, you get garbage, but if you pay them equitable wages, you get real work done. It's that simple.

Seems like you're painting with a broad brush. Design of HITs on Mechanical Turk is crucial. Implement gold standard questions per assigment to ensure that you're not receiving spam from a bot.

It turns out this is one of the core problems with Mechanical Turk that drives folks to MobileWorks instead.

In Turk, users spend so much time implementing protection against malicious workers and bots that they need to develop real expertise in crowdsourcing in order to be able to outsource their work effectively.

We've filtered workers more effectively up front and reduced the incentive to cheat substantially, so things largely just work.

I'm immediately reminded of Manna:

http://marshallbrain.com/manna1.htm

Let's hope it doesn't end up that way! The article only briefly mentions it, but we've worked hard to make sure that even if tasks are algorithmically generated for MobileWorkers, they have the opportunity to collaborate on task execution in a way that's much more similar to a virtual office than to being a cog (persistent worker-to-worker chat, managers to support and explain tasks, and more).

This makes a material difference in the quality of the final results, lets people enjoy their work more, and lets individuals take ownership over what they're doing.

Oh, I'm not saying you're heralding some kind of dystopian future, it was the headline more than the article that reminded me of the story. It does sound like you're doing good work.

Incidentally, much as that story does drag on a little in places, it does eventually present a more utopian part of that hypothetical future world -- one where post-scarcity has been embraced, and the technology is used to increase freedom instead of restricting it.

Add a couple more layers and you get... algorithms managing humans who control robots to clean for humans:

http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1

According to company cofounder Anand Kulkarni, the aim is to get the crowd of workers to "behave much more like an automatic resource than like individual and unreliable human beings."

Rationally, this makes sense. Emotionally, I find it repulsive. I'm not sure which impulse is stronger.

I feel like if I were involved in this company, I'd have to focus to some extent on providing a leg up to people in developing countries. Turning people into (potentially miserable) cogs in a machine would weigh on me after a while.

I'd have to focus to some extent on providing a leg up to people in developing countries.

Get them paying jobs which involve nothing worse than sitting at a computer screen and you're already more effective than, oh, approximately all charity and foreign aid. After you have the jobs you can even try to make them fun jobs, but un-fun jobs beat 3rd world poverty by quite a lot.

I don't think fun jobs are the goal. I think merit based opportunity is the goal.

Our goal is to get the crowd to behave like an automatic resource, taken as a whole, while providing a decidedly non-cog-like experience for individual workers. We want users to be able to trust the crowd, but we want people to behave, well, like people.

In MobileWork, workers work together in small teams under real names, have a dedicated manager and persistent chat, and talk and collaborate closely on work. Even if there is an algorithmic component to the work, the outcome is much more of a virtual office than the anonymous task machine that you're describing. This worker-centered approach is also a point of departure from just about every other crowd platform out there.

To tell the truth, we've invested considerable effort into developing an environment where we'd actually be happy working ourselves. Our studies show that this approach works: far from feeling like cogs, folks really do enjoy being MobileWorkers. See, for instance, our upcoming paper in IEEE, cited below.

There's a rational reason for this strategy. Our starting premise is that happy, motivated, and fairly-paid workers working in a collaborative environment will outperform anonymous crowd agents. Helping people with a leg up is the other half of that mission.

I don't want you to come away with the impression that I'm attacking what you're doing. Day to day you're providing a hell of a lot more opportunity than I am... and creating a lot more value at the same time.

I think the root of my emotional response might lie in an idealistic desire to see the opportunity for merit based latter climbing. Rationally, however, just creating a revenue source for people is a hell of a leg up.

Certainly, no offense taken :) I recognize that there's often a knee-jerk reaction when thinking about startups that attempt to address social issues in the developing world: it's not easy to separate out work that is ethical and impactful from work that is well-intentioned but harmful, particularly when the technologies involved are as new as crowd computing is.

You might be happy to know that we actually do offer opportunities for advancement within the system, and that it's one of our long-term objectives: there's a nascent system of skills and certifications that workers can train towards and develop, and workers can advance from doing very simple work to sophisticated work.

Eventually we expect that our workers will be able to develop enough certifications to transition up out of MobileWorks and into the traditional economy -- or better yet, to get to the point where they can build their own business around MobileWorks, as some workers have done. More on our philosophy here: http://mobileworks.com/fairtradework

Frederick Taylor would be proud:

http://en.wikipedia.org/wiki/Scientific_management

Interesting article, but somehow the comment section turned into a MobileWorkers commercial?

Reply on news.ycombinator.com