Hacker News Reader

Comment by bravura | Hacker News Reader

Ever since CrowdFlower (now Figure8) decided to terminate its SaaS-offering and focus only on enterprise, Amazon Mechanical Turk has been impossible to use.

CrowdFlower maintained statistics about the accuracy of individual workers. Additionally, they made it easy to include gold-standard "test" questions, which weeds out workers who are not doing the specific task correctly.

Without this sort of quality-control platform, mechanical turk is just unreliable crowdsourced-labor infrastructure. I understand that AWS doesn't want to build too much on the services they provide but, really, mechanical turk sucks.

I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered. It seems that everyone using turk has to reinvent the wheel. Why can't someone provide decent quality control over turk results?

View on HN

Because you're forgetting the other side of the equation, what sort of people are doing mechanical turk?

Are they 1:1 your peers, colleagues?

Or are they underpaid, most likely emerging economy countries, and very overworked labor that you're paying for?

You can browse to the mechanical turk subreddits and you'll hear the other side of the table that they spend 15-20-hours on a task to get what, a few dollars?

Pay more, get better results. The system itself is designed to exploit.

I have definitely upped the pay and not achieved better results. I don’t think it’s quite as simple a solution as you’re making it out to be. Realistically no matter how much you pay people who don’t have the medium to advanced skills to do the job for whatever reason, they won’t have those skills, at least in the short term.

Those skills are typically built through training anyway, but MTurk doesn't seem to have the necessary training pipeline.

How many of your peers and colleagues are regularly checking mturk for suitable well paying tasks?

Yep! The old saying "you get what you pay for" applies to Mechanical Turk like everything else. We tried outsourcing a project to Mechanical Turk and ended up spending more time on reliability and quality controls than it saved is in doing the manual work ourselves.

> I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered

For doing research, I think Prolific [1] is a popular alternative.

They include features like pre-screening based on demographic features (including participants' approval rates in past studies) [2], and attention check questions [3].

[1]: https://www.prolific.co

[2]: https://researcher-help.prolific.co/hc/en-gb/articles/360009...

[3]: https://researcher-help.prolific.co/hc/en-gb/articles/360009...

My startup, tasq.ai, was founded to solve this exact problem, as we have a live, successful product with all of the features you describe and more.

We're happy to have a conversation - feel free to reach out to me via my profile email or you can find our contact details at https://www.tasq.ai/

Same!

I created Cogmint.com ("cognition minting") to solve this problem for myself.

You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is. It's been a fun challenge to create simple methods of scoring similarity across different task types.

It's a side project, so don't rely on it for mission critical things, but I rely on it for some production tasks, so it's stable.

It currently supports classification (choose from a set of possible answers) and has beta support for bounding box task types. String input task types are coming very soon.

I'd love to see if it can help you out, I'll waive the fees: I'm not in it for the money I just like making things useful and reliable. Reach out and say hi!

We can find your site, but not your email. (Email field in your HN profile is always hidden; to share an email, put it in “about” on your profile.)

The problem of providing quality control is that there are a lot of edge cases; even known “high accuracy” turkers may have bad judgement sometimes, which means that every piece of data needs to be validated anyway, whether it be the researcher themselves or another paid contractor.

My undergrad thesis was to build https://tagbull.com, where we tried to have turkers validate the work of other turkers by breaking up a label into sub tasks, and getting multi-turker consensus on those before moving forward.

The main issue we ran into is that the incentive system is incredibly misaligned with the responsibility that the turkers have. It’s very difficult to build trust, especially with a crowd of people who haven’t signed contracts, and who face virtually no repercussions for doing bad work, whether intentionally or unintentionally.

> You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is.

Have you noticed problems that show up with questions whose answers have a bimodal distribution (ie. The gold standard question actually has two or more correct answers)?

In one sense, this is just a labeling quality problem with the 'gold standard' data, but to a lesser extent these same issues may crop up in the data being labeled when using similarity or clustering to rate or classify the workers and transitively apply that to the other results they produce.

Anecdotally yes it's a problem if two classes (button choices) are similar, resulting in two "top answers" for a given task. This seems most common for "yes/no" task types where there are only two options, and distinguishing between them is the hard part.

I haven't dug into the data on this across the platform but you've given me the idea to go see if I can find evidence of this, and see if I can improve somehow. There's only low hundreds of projects, so I might be able to find some that have this problem.

Sounds like a business opportunity?

Quality/metrics layer as a Saas with MT on backend, add a few percent to the price per work unit?

Sounds sweet. I bet a lot of companies relying on mturk built this for themselves and then sell a higher value service with better margins. You could build something right in the middle.

I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit.

I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.

Sounds a little meta! Managed managed labeling workflow. I do wish Amazon would do a better job with this, and I also have no doubt if a company like this sprang into existence and was successful, Amazon would clamp down on it in some way — whether by changing terms to disallow such a business, cutting off MT access to the business, or just copying it.

As AMT’s largest requester, we have longitudinal (~3yr) worker quality data for over 85% of respondents. Feel free to DM for details

How do you deal with respondents who just created their account?

Technical and Demographic user metrics are federated across the platform (we supply tasks to other labor markets online, essentially where bots and farms typically attack) so we leverage priors and shared similarity weights. Behavioral user metrics aren’t the only way to evaluate a respondent

Given that a business pivoted away from exactly this, when they had this up and running, I'd advise putting in some effort to find out why they did so first.

Mechanical Turk custom tasks are insanity as well. I was doing an object detection project and threw together a labeling UI. Integrated the Turk custom task, create and complete, but then I realized these tasks don't show up in the UI anywhere. You start implementing an entire UI for Amazon before you just bail and contract the one good Turk worker you've found.

CrowdFlower / Figure Eight is now part of Appen FYI

Reply on news.ycombinator.com