Mechanical Turk shutting down September 30

I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.

Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.

Long story short: Mechanical Turk saved my bacon.

Back in 2005, I was working a job at a small-town newspaper in a town I’d never lived before. I didn’t know anyone beyond the staff (as I was working layout rather than as a reporter), and I had gotten interested in the idea of doing Mechanical Turk for a few extra bucks. I thought there might be a formative scene of interested people doing this, so I started working on a blog for it. I briefly collaborated on it with another guy who put it on forum software because he didn’t know how to use, like Drupal.

That blog was called Turking.com, and it seemed like it was going well for a bit. We even got a mention on the AWS website. But after about two or three months, it was clear the initial excitement around the idea (and the initial work) had died down. (The work picked up later, but in clearly different ways. I don’t think folks were really “excited” about it after that point.)

If you want to get an idea of it, there was one capture on the Wayback Machine: https://web.archive.org/web/20051124231722/http://www.turkin...

The site had started to die out in part because my iBook suffered a catastrophic GPU failure and me, being a broke small-town newspaper employee, did not have the money to replace it. Plus, the community just hadn’t emerged like we had expected.

But then I got an email out of the blue: Someone wanted to buy my domain, which I owned outright, but they didn’t want to say who. I got contacted by a broker, and the exchange took place over escrow.

(The domain most assuredly was bought by Amazon and is managed by MarkMonitor.)

I didn’t get a ton of money from the deal, but I did get enough to pay for a new laptop. I had some regrets about selling it (in part because I originally built the site with someone else), but I was in a bit of a dire financial situation which that proved to be the starting point for getting myself out of.

That domain purchase was notably more than I ever made from clicking and classifying random pictures, that said.

The effort failed to find any areas of interest and the missing aviator was found the following year by a hiker. I wonder if there was any analysis after the fact to understand if the imagery actually provided any hints regarding the eventual crash site.

They are terribly monotonous tasks.

I can see why it's shutting down if its still the same thing.

Lllms could probably do everything there without rotting out minds for basically pennies.

Well it did work, in the sense that I managed to complete some tasks. I don't think I ever collected the money/credits back then though (simply because it amounted to so little). What I can attest though is that... it was debilitating. If you think your office job is boring then splitting it in way smaller tasks where you have no autonomy is absolutely terrible from a worker standpoint.

I initially was hoping to use it in order to work on providing a service over a dataset but understanding first hand what it takes to make it happen made me stop. It radically changed how I saw supervised learning since, and sadly not in a good way.

It appears now we can get along with just a single “artificial.”

The problem as I see it is verifying the work is not done by AI. If it was possible to verify somehow that the work is definitely not done by AI I think there would still be a market for this. But the data just cannot be trusted.

An increasingly big part of my job is to try and weed out workers who use AI and it's not an easy task at scale because generally workers gain trust with manual work and then there is degradation over time.

We had pretty good luck in the end, but getting there produced a somewhat large app on our side that would manage the whole process, including but not limited to asking for multiple responses, comparing them to each other, finding consensus, and determining which users would consistently produce bad responses and stop them from responding. We got to a confidence that about 85-95% of the data was correct, which was good enough for the company.

Through that process, I learned a couple things about managing mturk, primarily about how changes to the cost-per-task would change the process. Initially, we thought that price would be a quality knob, but quickly learned that price was a speed know. The higher the price the faster the tasks would be taken and completed. Quality did not change significantly as the price went up or down.

Overall, I still have fondness to mturk, but it was really bare-bones experience that needed a lot of work to get working effectively.

https://arstechnica.com/information-technology/2017/11/expen...

Instead of partying in college, my friends and I wrote a script to complete the tasks. We took over university computer labs to run it on a bunch of different computers. We made a few thousand bucks. Good times.

I think the weirdest thing I had to do was scan through social media images... so weird seeing into random people's lives.

I did it in the late 2000s or early 2010s.

The only use I can think of it these days is for social scientists who need to gather judgements from bona fide humans.

This will be devastating for a lot of workers in low-infrastructure countries.

Incentives work both ways. If the exchange rate of human labor to capital falls to zero, then human workers will have no choice but to exclude capital from the equation.

No jobs, no money, no billionaires.