Mechanical Turk workers are using AI to automate being human
techcrunch.com
techcrunch.com
Comments from 2 weeks ago.
This research study is using a very simple task, summarization, that is well solved by AI (to the point that everyone points this out as the first use case). Of course, the MTurkers are gonna paste it into ChatGPT. (Also n=44)
I remember worker forums sharing scripts to query amazon for the expected results that would get you paid. Amazon put up a bunch of very lucrative test jobs, but you only got paid if your answer matched the majority. The majority were automating, so unless you used the sometimes incorrect answers, you got nothing.
you are only counting the human labour involved in the creation of value.
what about the capital?
Would you, as a hired gold miner, be able to accept zero pay for your work, until the gold is sold (then you got the full price of the sold gold)? What if you starved between the time you mined it, and the time it takes to sell? And how would you have gotten the equipment to do the mining in the first place, let alone the ownership of the mine.
The previous statement was in regards to the extraction of value - modern markets are such that the best(/fastest/easiest/etc) way to make money is to be at the top of the value chain and let your capital do the work. This is a self-reinforcing feedback loop, enabling you to use your increasing gains to find other forms of leverage (in your example - buying the mine and the equipment, setting up commodity trading positions, etc) and keep on ratcheting up the leverage. One inevitable outcome is for you to have the opportunity to use this leverage against the people doing the actual digging up of the gold.
If value was derived from capital itself, then a pile of tools or machines would spontaneously generate wealth without the application of human labor, and idle money in a bank would somehow enrich society.
The standard "how come patrons at a bar pay $10 for a drink but I only make $15 / hour when I serve twenty drinks an hour" argument.
The answer of course is because you didn't create the bar, you didn't secure all the necessary supplies and amenities to run it and ultimately you're not liable for what happens to it. In essence if the bar loses money each month you're still owed a paycheck.
And ultimately that's why raw materials and content DON'T have inherent value (e.g. dug up gold sitting in a mine shaft, the best song ever buried in your notebook).
What they have is POTENTIAL value if the infrastructure exists to get them processed and to the locations where people want them and if people are willing to put skin in the game to bet it will pay off.
And that need for infrastructure is the reason that argument always falls apart.
Co-opts are about the best feasible answer we have now, but it always seems to be the case that suddenly people have far more capitalistic notions about running a business when THEY become the majority risk holder for a new venture.
Only until the bar stops existing. You're on the hook for business continuity whether you're the employer or the employee. Lex Fridman and Richard Wolff discussed this: https://youtu.be/o0Bi-q89j5Y?t=2984
Co-ops are indeed the closest thing we have to escaping capitalism in America. However they don't grow very quickly if at all, for the very reason that they aren't efficient at extracting value to fuel growth.
At which point you are STILL not responsible for any liability. For example if the bar gets sued, you don't lose your home, car, etc...
And in most countries, would receive unemployment that was at least partially paid for by the business in advance.
Notice again though - just like the gold mine - without the infrastructure you can't extract any value at all from your ability to pour drinks.
Wolff is about as bad faith as actors get.
Uh, you lose your car and home when you can't pay for it due to the bar stopping existing. So you are liable in the sense that your entire income disappears due to this event. It takes a certain kind of boneheadedness to not see this.
> without the infrastructure you can't extract any value at all
I'm not sure what your point here is, all you've done is point out the obvious reason this system can even exist in the first place. Congratulations? If it weren't the case, people would just work and get the money directly and we wouldn't be having this conversation.
I'll just ignore your baseless/citationless assertion at the end there.
It's always painfully obvious when someone has never had to run even the microist of businesses.
They literally can't perceive the idea that most business owners are taking a huge risk and putting in far more hours and effort than their employees who are free to just walk away any time a better offer comes along.
If you're doing it right, you don't do that. That's why corporations exist, to separate personal and business assets. If your corporate veil gets pierced, that's because you don't know what you're doing.
> far more hours
I mean, if you've ever been at a startup it should be patently obvious that the owners (the board) are not working anywhere nearly as much as people in the office who stay late and many times sleep there.
> free to just walk away any time a better offer comes along
You're absolutely able to do this as a business owner. Businesses get sold and liquidated all the time.
If no one promoted musicians, organized tours, etc, then buskers would be the extent of what’s possible.
If one person had to mine, refine, transport, and sell gold that person would do each of those things more poorly than a group of people who specialized.
You are getting at something, but it’s still unarticulated at least from my perspective: large systems of specialized workers produce the opportunity for people to construct and/or capture and maintain chokepoints. With undue control of these chokepoints a person can profit inordinately.
This is part of why anti-trust regulations exist: when the entirety of a class of production is under one coordinated control, that is the ultimate chokepoint. But what you are talking about is a smaller version of that.
ultimately you need to get away from being reliant on a salary and become a capitalist yourself. otherwise you're just one of billions that are a few bad breaks away from starvation.
You assume I'm not a capitalist. When did I say I wasn't?
Based on the average level of competency I see (There are definitely exceptions), I would disagree...
That would be kinda funny to me tbh
Even printed references can be based on wikipedia now
E.g. someone writes a made up citogenesis article on Wikipedia before the xkcd comic was drawn.
Note also: “Since the end of atmospheric nuclear testing, background radiation has decreased to very near natural levels,[3] making special low-background steel no longer necessary for most radiation-sensitive applications, as brand-new steel now has a low enough radioactive signature that it can generally be used in such applications.”
In particular small and/or dying human languages. Languages carry with them an immense amount of information, from all the precious human experiences that have shaped them.
You might argue that if it's old and still common today then it must have guided the people who followed it well in order to survive so long, engaging in a kind of natural selection of ideas, but the thing is that ideas can also fall prey to suboptimal equilibria and all sorts of other evolutionary weirdness and badness. Like, the success of an idea is not directly related to how good the idea actually is for the people that follow it. As long as it isn't bad enough to kill off all its followers, an idea can spread by many other means. Maybe it's better at spreading, through appealing to our baser natures or cognitive biases, maybe it's good at locking itself in through thought stopping cliches and fear of reprisal and stuff like Pascal's Wager/Roko's Basilisk. Or maybe it's really good at setting up interlocking peer pressure and network effects and social enforcement of continuing to believe it. Or maybe it's better at suppressing other ideas. Or maybe it's better at creating a group geared toward conquest or evangelism, but actually living under it sucks otherwise. Maybe it's just really beneficial to a small group of people in power and so they work really hard to spread it. Not to mention, it only had to compete against the actual ideas that existed contemporary with ut in history and were strong enough to create evolutionary pressure, so if a new idea comes along about how to organize society or whatever, we don't actually know if it will automatically win out just because it won out against other different ideas in the past. What would we think an idea is unimprovable or optimal just because it is old? The long term existence and popularity of an idea or tradition doesn't necessarily make it good at all. Would you say that the Bible's injunctions against e.g. divorce or homosexuality or women being able to teach and hold positions of authority are encoded wisdom? The most valuable data there is?
I'm by no means an actual positivist, which is an incoherent position, but I am a pragmatist, and I think the only way to actually figure out if a tradition is worth anything is to look at its reasoning, its assumptions, its history, and its effects in the world.
We know the data survived a narrative/biological co-evolution distributed process spanning thousands of generations over many millennia. It doesn't make it perfect, but it makes it pretty darn good with high likelihood, especially when compared with narrative complexes invented yesterday, which have yet to withstand the test of time and which show measurable signs of dysfunction.
See also the Lindy effect, https://en.wikipedia.org/wiki/Lindy_effect "the future life expectancy of some non-perishable things, like a technology or an idea, is proportional to their current age"
This is fallacious. Already responded to this at length.
> which have yet to withstand the test of time and which show measurable signs of dysfunction.
Unlike the older traditions, which manifestly have horrible dysfunctions and consequences for a lot of people in a lot of areas. Should we never try to fix and improve those failings because we "dont know what might happen"? Or is that unknown worth it to actually have a shot at really making things better? Why are we acting like we can't have a reasonable idea of how good or bad an idea is based on reasoning through the idea, why is the null hypothesis that any societal change could cause civilizational collapse when there's no evidence of that? It's like the people constantly claiming gay marriage will cause the collapse of western civilization. Also, why are we weighing the known horrors of our inherited traditions so lightly you barely even acknowledge them, so much more lightly than the vague, unknown disasters of... civil rights or gay marriage. Who knows what disasters letting women vote or have their own bank accounts or get jobs might bring! The horror!
> See also the Lindy effect, https://en.wikipedia.org/wiki/Lindy_effect "the future life expectancy of some non-perishable things, like a technology or an idea, is proportional to their current age"
Lasts a long time =/= good.
In other words, to get a better grip on the what we're trying to achieve with any of this.
The reason languages are so precious, is that it's deeply imprinted in them much of what their speakers considered valuable. That can give us a new perspective on and deeper understanding of the things we find valuable.
Really? My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results, but happy to see some counter examples
It’s a mass energy balance. The inputs equal the outputs; garbage in, garbage out.
You'll never hear AI from Apple announcements, this contrast sharply with the few other large tech companies..
Kinda like crypto but on smaller scale.
Y'all cheerleaders are going to create an underclass the likes of which you've never seen before. I hope they take up torches and pitchforks
I live in a country where the government is actually calling for an INCREASE in work. When in history have we ever decreased the workload with the reason "well we don't need to produce any more"?
Yet all this tech stuff does is create even more. Working hours are up and productivity is down, and purportedly nobody knows why.
I want my 4 hour workday, goddammit
I don’t think work will go away, it will just change, maybe too quickly for people to keep up.
Post AGI nobody knows. It’s debatable that humans will still have a place in an AGI world. But that’s also much further away than people seem to think. I don’t expect to see it in my lifetime (next 40-60 years).
or those who become unneeded would simply starve to death, or have to compete to get the last vestiges of available jobs.
I know which is more likely, from looking back at history.
Where is this mass displacement of which you speak?
Which companies are firing en masse and replacing with AI?
More locally, the local McDonalds recently remodeled, reducing the customer counter to something like 1/8th the size and installing terminals (not AI but automation), and I've seen images online of fast food testing AI for ordering, particularly over drive-through.
My friend got a marketing director job using a ChatGPT-generated marketing strategy (they gave her an impossible deadline and so I suggested she try it), and now she uses ChatGPT to generate marketing copy instead of hiring content writers. Someone else I know uses AI to SEOize very technical text to something search engines like at like 10x the rate he used to do by hand.
As a solo and formerly senior SWE, ChatGPT is like having a junior developer on call to do my research and get me preliminary results. Stuff I used to have to rely on my team or google-fu to answer, now I just ask ChatGPT.
You're sticking your head in your sand if you think it isn't coming. Just think about every one of those optimizations that AI offers, typically you had to have a human being do it before.
I dont think the doom and gloom scenario being sold as the AI apocalypse is as bad as it is being portrayed.
Who? Be specific. This displacement is happening rapidly, it should be easy to point out.
> The trucking industry is dumping tons of $$$ into automated AI trucking. Tech companies are experimenting with drone delivery instead of hiring drivers. Rideshares experimenting with self-driving taxis. The list goes on.
How many truck drivers have been laid off from this technology which hasn't been deployed widely / left the experimentation phase?
> she uses ChatGPT to generate marketing copy instead of hiring content writers.
So far the only real competition you've mentioned. And yeah, language models are good at writing low value language. I have yet to hear about mass unemployment of content writers however. These are people who will upskill/shift into other marketing roles easily.
> Someone else I know uses AI to SEOize very technical text to something search engines like at like 10x the rate he used to do by hand.
And this is a bad thing?
> As a solo and formerly senior SWE, ChatGPT is like having a junior developer on call to do my research and get me preliminary results.
You say this as if companies will stop hiring juniors. They won't. The world will simply write more code with more productive people.
> You're sticking your head in your sand if you think it isn't coming.
The same has been said during every technological shift. Mechanization will result in mass unemployment of laborers. Email will result in mass unemployment of mail carriers. Through many cycles of technology, the human race has never allowed slack in the workforce to go unused for very long. We could have stopped at the steam engine and accepted the quality of life we had, instead of repurposing displaced workers and marching forward. This won't change until we have true AGI which we are nowhere near.
AI will take some jobs, but there is another risk. It's far more likely that they "de-skill" a certain aspect of work so it becomes less creative and more mundane, more easily replaceable. The goal is to turn creative work into something like a position on an assembly line.
The problem isn't technology, it's in who it benefits. Productivity gains could benefit everyone. We could all get more time off, for example. Instead, those gains make a small number of people extremely wealthy and everyone else more miserable.
The average person active on HN could do this. They could move somewhere with extremely low cost of living and work a remote job. But almost nobody does this. Instead people choose to live in incredibly high cost areas at well paid jobs, chasing ever higher quality of life.
It's not some boogeyman forcing this. It's just human nature.
I just freakin said who. That is very specific. What are you looking for me to name Jessica, former content lead at AcmeCorp?
> How many truck drivers have been laid off from this technology which hasn't been deployed widely / left the experimentation phase?
None, yet.
But you are missing my point. This stuff is beginning to happen. Much has yet to happen. You are betting that it won't happen, and I am saying you are wrong. Very little of this will empower people where they are, at the lowest rungs of various corporate ladders. But it will empower their taskmasters, who will happily shed them with the same bullshit excuses they've always given: "Not my problem" and "Sorry, it's just business."
Since you're so damn optimistic about the future, what are these jobs that these people will be retraining to? That don't exist yet. Please share just a little bit of your visionary insight.
How about a company or an article?
> But you are missing my point. This stuff is beginning to happen.
I'm not, you're missing mine. You believe they are beginning to happen. But you tell me they haven't happened yet. There is nothing substantive to justify this fear. History certainly doesn't. That's my point.
> Since you're so damn optimistic about the future, what are these jobs that these people will be retraining to? That don't exist yet.
Imagine that, the future creating new jobs for people. Not as if that has ever happened before. We're all still toiling in the fields aren't we.
If LLMs don't flame out it will be like the dawn of offshoring except wider scale and without the upside of poor people in other countries making more money; just more wealth concentrated in the hands of ever fewer people.
Technology is a double edge sword in every way.
Jobs which are specifically about being a human seem like they can't be automated, like actors, waiters, athletes, sex workers.
I sincerely doubt anyone would watch or care about fake AI generated sports.
Well, people pay for fake sports, and pay for fiction (i.e. fake stories) in movies/TV/books.
Not much of a stretch that a competent AI could generate good enough content, given the sludge currently making money out there ...
What AI is likely to do is distract people from the continued accumulation of wealth and power. Reinforce the notion that it's "fair" (everyone else can be automated after all). And provide a cap to your wages (if the machine could do it with 50 MWh of power, then your salary will logically have to be lower than that, even if the machine doesn't actually exist YET).
The cotton loom didn’t present itself as a means of accumulating wealth and power, but yet it did. Why would you pay a group of people that much to weave when a cotton loom can do the work of 10?
Anything that makes things easier will reduce the number of jobs.
Smart phones improved billions of lives and displaced a relatively small number of jobs. Global shipping routes decimated hundreds of towns and cities, many of which still haven't recovered decades later, with 90% of the benefits going to 1% of the population.
There are massive ranges possible on the winner and loser sides of any new technology equation. I'd argue that widespread AI will cause broad harm and very concentrated gains, which I think will be very bad for society on the whole.
I wonder if we'll reach the point where "AI" ends up with the same negative connotation as "Made in China".
Paul McCartney has had to repeatedly point out that the new song they are producing is John Lennon's real voice, and AI was only used to separate and clean the tracks up.
Even still, there are real musicians out there that will learn how to use AI to generate musical ideas that are new and fresh. They'll ironically have to be in the closet about it, lest people just assume the musician didn't do anything beyond prompting for musical output.
ML and LLMs in particular are just fancy maths/statistics. Yes they are immensely useful, but also 1. overhyped / overrated, 2. facing the problem of in-breeding.
AGI will work nothing like today's LLMs. AGI will stare at your question, and instead of answering, ask you: "why do you want to know that?". You can't have intelligence without curiosity.
We will no longer be the dominant species.
If the AGI is anything like us, it might keep us around as slaves.
Not immediately, but give it 100 years. We’ll be the frog in the pot of boiling water.
AGI final form is giving non-answers a-la stack overflow.
That's just garbage propaganda from the hype men. I don't buy it and you shouldn't either. We don't live in a sci-fi fantasy novel.
The real end game for "AI" is power for the few, with little need for the plebs (us), so no need to share any of that power. That's what our current social structure will give us, and "AI" isn't going to change that structure.
If a hypothetical machine could do your job, well you better do it cheap enough that we don't start researching that. It's like UBER, the taxis will drive themselves soon enough, so you have to work for no pay for now, to usher in the future where you will make money. Nevermind that it never happened and people are still working slavery wages.
Pascals Wager basically.
It will make a lot of money for existing rich people, sure.
But will it help a family pay for rising medical, housing, and food costs? Absolutely not.
It will mainly serve to optimize and automate, but under a profit driven system, optimizations only serve to extract more wealth, not distribute it.
Considering the current economic system and state of social safety nets, I have to disagree with you. I bet the widespread adoption of ML models will only increase the Gini coefficient by doing to knowledge-workers what automation did to factory workers. The middle-class will be hollowed out.
Instead of trying to slap a technical fix onto that problem, we could be working to address root causes.
Superintelligence is when you can make all the money because all the problems are solvable. Right now only some are.
Also if self driving cars are everywhere then we won't have AGI, because real AGI largely does away with the need for cars.
1. Give a sample of mechanical turk workers the same data to classify without telling them
2. Once submitted, look carefully at the ones they get wrong
3. If the turk worker gets similar ones wrong to other turk workers, they're probably not cheating
4. If the turk worker is frequently wrong on questions other turk workers all got right (e.g. if 99.9% of other workers got it right) then they've likely used AI to generate their answers and then fiddled around with a few afterwards to make it look human
That presumes they're fiddling with a few responses for the human touch. If they're submitting raw AI output that should be easily discoverable by running similar models against the same data (like detection of chess cheating compares games against moves form several engines, not just one).
I think that would work, unless everybody uses AI. If that latter were suspected, get a group to label data in person (where you can be 100% sure they can't cheat), then use those answers as the basis for comparison.
City map makers didn't want people to copy their hard work, so they'd add a short street or some other feature that doesn't actually exist in reality.
What dog whistling drivel. That dude's trying so hard to use fancy words and walls of texts to defend his racist views without stating them outright. This is coming from a person who previously said:
"One quality that's a really bad indication is a CEO with a strong foreign accent. They must be clueless if they haven't gotten rid of their strong accent."
Already going mask off in public and later doubling down on it in his blog. Just imagine what his views are in private if he goes to those lengths to try and fail to morally justify them.
Edit: I found the context around it, at least. Seems perfectly reasonable. [1]
"Recently Inc published an interview in which I said we'd noticed a correlation between founders having very strong foreign accents and their companies doing badly.
Some interpreted this statement as xenophobic, or even racist—as if I'd said that having a foreign accent at all was a problem.
But that's not what I said, or what I think. No one in Silicon Valley would think that. A lot of the most successful founders here speak with accents.
The case I was talking about is when founders have accents so strong that people can't understand what they're saying. I.e. the problem is not the cultural signal accents send, but the practical difficulty of getting a startup off the ground when people can't understand you."
Per a comment above, that for a long time now they've been sharing scripts to get the same answer so they can get paid, this indicates cheating.
It seems we will be soon drowning in an endless sea of autogenerated noise. For all the talk of X-risk, this is the imminent tangible risk we can perceive and reason about.
If we can't find solutions to the problems that primitive AI will create, then solving the harder problems of AGI/ASI might become moot as we self destruct before we even get that far.
FYI, my summary of many other potential road blocks and stumbles we are likely moving towards in the near term.
Pay people extra money for the privilege of having them fulfill the tasks the way you want them to.
GPT-4 going toe to toe with the experts who set the benchmarks, way overperforming the crowdworkers https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro...
GPT-3.5 outperforming crowdworkers https://arxiv.org/abs/2303.15056
More money won't stop people from automating whatever they can. Especially when the automation provides such good results.
Humans go insane if we lack sufficiently varied sensory input. Dreaming doesn't really help with that, and I recall a recent paper that showed something similar for AI.
Yes, such as an office setting, or the office setting of a contractor. Which are both options.
-- h2g2
At what point do Amazon just switch off Mech Turk and leave it to AI?
I guess it's best to make hay in the interim
I dunno who would continue to use it, rather than use the AI engine directly, but there could be a market for it. But they'd certainly need to make certain clients knew about it, since one major use of MTurk is to do surveys and studies whose whole point is to sample human beings.
That's exactly what they meant.
As others have pointed out, mturk always had workers using automation, and the researchers set up a task that is an obvious application for LLM's, who's output makes it difficult to different humans generated output, and isn't even a good use case for mturk (since the output is free form text that would have to be validated by humans rather than an automated quorum check).
This is simply an insult to science
Amzn is literally bad for everything on the planet aside from commerce - they are the seed of the Ferenghi