Hacker News Reader

Comment by ftxbro | Hacker News Reader

Also "the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk."

It's interesting that the best available MTurk Master crowd-workers located in the US are paid about six cents per task.

View on HN

I guess my surprise is that the machine is only 20x cheaper than the cheapest human available.

It’s worse because a US-based MTurker is far from the cheapest human available.

Hey, with any luck, running one of these bots with the capability to replace me will cost just as much as hiring me.

(Ha! As if. Costs are only going to come down)

Against the cost of living in the US, workers in the US are about as cheap as they come, second to maybe sweatshop workers in free trade zones or slaves in Dubai.

ChatGPT costs the same, regardless of the difficulty of the task, which is not the case for humans.

Given how badly the humans performed, I suspect a lot of them are either not bothering to read the prompt or using a worse bot than ChatGPT. Not surprsing they are so cheap.

If you tried to pay people to do it properly then presumably it would cost much more.

Given the low quality results of the humans, I suspect they were paying too low and the workers were giving low quality answers. I wonder if they should have paid more per task to get a better comparison.

It doesn't feel surprising that GPT is cheaper and more accurate than a human who is randomly clicking buttons. Custom software can already do that.

When will we see a form of "arbitrage" where someone uses ChatGPT to do the work of an MTurk and pockets the difference? Will that lead to MTurk prices converging to ChatGPT prices?

Even before LLMs, MTurk has been in a war with the botters, and my understanding is that the MTurkers have to periodically do captchas but many still use bots or various automated tools or utilities. The LLMs and especially the multimodal ones will only make the bots even better and break the captchas even harder. MTurk's business model specifically wants human workers categorized into fine grained categories so they will try to keep fighting the bots however they can.

Captchas are futile. I remember many years ago when I was in highschool, I came across a shady job posting on an eBay listing. The work involved solving captchas. There was a web page they set up that just displayed one captcha after another, and it was your job to sit there solving them.

I didn’t do it because I can’t imagine a worse hell than having to solve so many captchas, but it opened my eyes to how creative botters can be.

I can imagine someone doing a similar setup for mturk, where all you need to do is remotely solve a captcha on your phone every few hours/minutes while ChatGPT does its thing. I also wouldn’t be surprised if that already exists.

Is there such a thing as a captcha in the age of GPT4?

Maybe openAI put safety barriers on their model... But surely the botters have a model without that by now?

If you have to select the matching images from 9 images and you are feeding each of those images to GPT4 and asking "is this a <TARGET>?", I don't think you're going to get a response back to complete the captcha in time (e.g. <30s). This will change as the models advance.

The per annotation cost of simple image tasks in MTurk can be as low as $0.0003

Reply on news.ycombinator.com