33-46% of workers on MTurk used LLMs in a text production task | Hacker News Reader

33-46% of workers on MTurk used LLMs in a text production task

arxiv.org

189 pointsputtycat133 comments

So mechanical turk, which was a robot that actually contained a human, is actually a robot that contains a human that contains a robot.

We have achieved complete ouroboros.

And the final robot has millions-billions of humans fused using linear algebra.

And the humans are trillions of tiny nanobots (after a there was a "Grey Goo" event billions of years ago) that formed a hivemind that isn't even aware of all its own parts. :P

https://users.ece.cmu.edu/~gamvrosi/thelastq.html

It's robots all the way down!

We call that reinforcement.

It’s like actors in movies copy real world people who copy actors in movies.

RLHBARF - Reinforcement Learning from Human But Actually Robot Feedback

Uh, I think this should be a thing considering we have LLMs training other LLMs. so we end up with this BARF loop.

There is an actual interesting question here if BARF loops can create new synthetic knowledge, or if humans need to be the 'spark of novelty' in that loop to drive it to new knowledge.

I love your acronym formation there.

There's another thread running at the moment about BARF degrading the machine's performance. I don't have enough information to be confident, but it sounded plausible and likely.

if BARF becomes a thing I really hope I'll be credited for it haha

Q: Does art imitate life or does life imitate art?

A: Yes

no it is only an ouroboros if biting its own tail -- this is more of a cyborg centipede situation

Just wait until the LLM extensions are better. The future looks like this:

1. You use MTurk to brainstorm new product ideas

2. The human figures out your company and tasks the LLM with the brainstorming

3. The LLM hits you up for a chat about your company's future :)))

Watch some videos of various factory production lines, or ag planting for example. Machines that feed stuff to humans that may feed more machines.

There's a great, if rather old, word for these machines: "tools".

Or perhaps "ourorobos" :)

> We have achieved complete ouroboros.

That's a generous / romantic take. For me, the Human Centipede of online text generation is now a complete circle

Awful, but strangely appropriate. Considering all the "I don't care about quality; just dump a pile of text" uses we're seeing.

true orourboros comes when it's undetectable which level of the turtle you're on.

33-46% of students in the local high school used LLMs in a text production task. I mean, they actually did.

[1] https://www.fastcompany.com/90841387/gpt-3-chatgpt-high-scho... [2] https://www.theatlantic.com/technology/archive/2022/12/opena... [3] https://www.nytimes.com/2023/01/12/technology/chatgpt-school...

The Turing Syndrome speeds up!

Turing Syndrome (Like the Kessler syndrome) is when the amount of AI generated data on the internet surpasses human generated content to the extent that it will eventually make it impossible to distinguish between the two

That's not how it's actually going to play out.

As the AI generated content becomes dramatically overwhelming in scale, the human content will become increasingly easy to spot (and there will be multiple cues to the human content that make it fairly obvious). There will be a crossing of the two along the way, in regards to the amount of content generated, a relatively brief time where it will be difficult to tell which is which.

The more AI content there is, and the more it advances, the easier it's going to be to play spot the human. The time in which it'll be hardest to tell them apart, will be in the middle frames rather than in the later stages.

Because the human data will become "stupid" in comparison, is that the reason?

I don't understand why spot-the-human will become easier otherwise.

The human content will contain the n-word or other shibboleths.

This also doesn't make any sense, as it would be much cheaper to use local LLMs to generate text, making it likely to see generated toxic language.

The "n-word" or any arbitrary other shibboleths may guarantee you didn't use OpenAI services, but they don't have any sort of lock on LLM technology and it's trivial to ask non-controlled ones to use any you like.

AI generated text is still that even with human shibboleth decorations added

Deep irony to use such a dehumanizing word as evidence of humanity.

Its easy to sprinkle a few n. or other words onto AI generated content.

Like Kessler though the problem isn't to detect it but to get anything through. We might not see human content for a few decades before we've sorted it out.

Exactly! I just point to the root lack of differentiation as intractable

As in you can’t sort it out…just like so far there’s nobody that really knows how to clean up space long term

"As the AI generated content becomes dramatically overwhelming in scale, the human content will become increasingly easy to spot (and there will be multiple cues to the human content that make it fairly obvious)"

I feel like you're not accounting for the amount of that AI content that will be deliberately and intelligently intended to masquerade as human. Every signal you can think of and may start using is a signal that intelligent humans can and will forge. And they'll get the ones you didn't think of either, because that's their job.

I don't even have to hedge or qualify this prediction, because we have decades of experience with people already forging every signal that is technically possible to forge, and pouring very substantial effort into doing so, right up to having dark businesses that provide these things as a service. "Forge all the signals that are technically possible to forge" is a product you can buy right now. For example, from 13 days ago: https://news.ycombinator.com/item?id=36151140

I don't know what signals you think you're going to see, but I'll guarantee A: yes, you will indeed see them because there's a lot of degrees of competence in the use of these systems (I still periodically get spams where the sender sent out their template rather than filling it in with a mail merge), but B: those will just be the ones you notice, not the entire population.

Rather than a binary decision, I like to measure with "how much information does it take for me to detect a forgery". For instance, things like modern architecture renders can absolutely fool me as being real at standard definition, but at 4K they still struggle (too precise, too clean, even when they try to muss it up on purpose). I need a few paragraphs of GPT text to identify it conclusively, and that's just the default tone it takes. Ask it to give you some text in a specific style and I don't know how much it would take. For all I know I'm already missing things. You're probably already missing things you didn't realize.

ChatGTP has a number of topics it will refuse to discuss so any writing on those topics will be more often than not human generated.

As I mentioned in another post, OpenAI does not have any sort of lock on LLMs. There's already plenty in the wild and they're only going to get better. Already the commercial AI companies are expressing concern about how much faster the truly open stuff is developing than their stuff. The companies can still throw better hardware at the problem somewhat more reliably but that's not much of a moat; it's still commodity hardware, it's just a bit much to drop on just being able to LLM better if you're not being paid to LLM.

Do any publicly available services like this offer a uncensored bot? I have models I can run on my laptop without censorship but they aren't very high quality.

I know they can exist but from what I can get my hands on it seems saying something offensive is a way to prove you are human for the time being.

Llama 65B exists and you can just run it on a cloud GPU instance.

Ahh thank you, still that has a cost, legal and technical bar to cross but seems to meet the criteria.

Yes, I do agree with you that it is nowhere near as trivial as it is to just pop open ChatGPT and ask it something right now, and at the moment, that matters.

I'm looking more long term, like, a year out, five years out, and on that time frame I think it's very reasonable to expect to see ChatGPT-levels of quality being completely commoditized. OpenAI or whatever commercial entity may by then have the next level AI, but there's definitely a threshold where the combination of LLM + human ingenuity gets to the point where there are effectively no signals of humanity left.

Moreover, I'm running on the theory that detecting LLMs is in fact not an arms race, but clearly ends in a win for LLMs. Basically, there exists text that simply looks human. It doesn't matter what level of superhuman AI you throw at it, it simply is in the space of "texts that a human was likely to generate". Once that level is attained, it won't matter that there is something that can generate superhuman text quality or something.

(Or, put another way, if your clever "ride the AI wave" startup plan is "create a startup that can detect LLM usage", I advise you strongly to wargame out your 1 year and 5 year futures for that tech, especially in the face of deliberate attack. I would not be optimistic.)

Or maybe the complete opposite, as it will only be learning patterns created by AI, producing less variance at higher temperatures, ultimately overfitting onto the same bland jargon.

This is more like cousins having a baby with 3 nostrils

And a few extra fingers

^^ This comment was generated by A.I.

Neal Stephenson made some interesting predictions about this ~a decade ago

Edit: Anathem actually came out 15 years ago

Side topic: Anathem is actually my favorite book of his. I feel like it's under-represented in a list of top NS books.

I forget, how does this come up in Anathem?

I remember similar notions in his novel Fall — where the internet is so full of fake news and sponsored content that people need additional services to filter out useful information

(Syndev = computer (syntactic device), reticulum = internet)

> Early in the Reticulum—thousands of years ago—it became almost useless because it was cluttered with faulty, obsolete, or downright misleading information,’ Sammann said.

> ‘Crap, you once called it,’ I reminded him.

> ‘Yes—a technical term. So crap filtering became important. Businesses were built around it. Some of those businesses came up with a clever plan to make more money: they poisoned the well. They began to put crap on the Reticulum deliberately, forcing people to use their products to filter that crap back out. They created syndevs whose sole purpose was to spew crap into the Reticulum. But it had to be good crap.’

> ‘What is good crap?’ Arsibalt asked in a politely incredulous tone.

> ‘Well, bad crap would be an unformatted document consisting of random letters. Good crap would be a beautifully typeset, well-written document that contained a hundred correct, verifiable sentences and one that was subtly false. It’s a lot harder to generate good crap. At first they had to hire humans to churn it out. They mostly did it by taking legitimate documents and inserting errors—swapping one name for another, say. But it didn’t really take off until the military got interested.’

> ‘As a tactic for planting misinformation in the enemy’s reticules, you mean,’ Osa said. ‘This I know about. You are referring to the Artificial Inanity programs of the mid-First Millennium…’”

Which fairly closely mirrors the conspiracy theory about antivirus vendors...

This sounds more inspired by trap streets than AI data pollution, though I suspect the people trying to fight generative AI will eventually resort to trap streets anyway, and wind up producing the same outcome.

Oh cool, I haven't read Fall but it's similar in Anathem

The reticulum (aka internet) is filled with software generated crap. Initially most of it is blatant and easy to filter out, like an over the top "but cheap v14gra" spam email.

Eventually the companies that make the crap filtration software get into an arms race with each other, they realize if they can generate their own crap that their competitors don't detect, it'll give them a competitive advantage. The reticulum becomes filled with "high quality" crap, eg text that's ALMOST correct but wrong in subtle ways. Imagine a wiki article about pi where everything is correct except the 9th digit, or an op-ed with slightly flawed logic.

Eventually crap goes beyond text, and the reticulum starts to see deepfaked images and videos that parallel news articles. One of the jobs of the ITA is to try and find the signal in all the noise.

I think this is probably a poor metric for it. Mechanical Turk is full of people trying to race their way through surveys to make a few pennies. So there's a motivation to just create some output that can't be automatically filtered out, like missing a 'please select D as the answer for this question' attention test, while otherwise just going as quickly as possible. It's a dream scenario for any sort of automation, no matter how crude. It's one step above MMO gold farming.

Tangentially related, Mechanical Turk and its various less well known clones which come down to the exact same thing, are increasingly the status quo for social science studies. They keep making really shocking discoveries with like 99.99999% statistical certainty given the huge sample sizes these services enable. Kind of weird how often they fail to replicate.

The fact that MTurkers are paid makes them unreliable for reflecting genuine human responses.

That they can’t replicate indicates that the sampling population isn’t consistent representation of population demographics and so there’s no real signal there that can be applied to a general population like most failure to replicate studies

Not paying would be a heavy bias, too. Do we need something like jury duty?

The person you replied to was being sarcastic, but you're still right.

Sociology's reliance on MTurk is absurd but most professionally run polls are now the same thing, just dressed up to look nicer. This is why polls keep reporting lots of weird results which are then taken as absolute fact by researchers and pundits.

I saw one the other day from a professional polling firm in which exactly 7% of "the public" said they had attended a protest, no matter what the protest was about. They asked about six or seven things people might protest about, and for every single one, 7% of the panel said they'd been on such a protest. Taken at face value it led to nonsense like concluding millions of people had attended protest marches about CDBC and 15 minute cities.

Unfortunately, the panels these firms use are extremely unrepresentative of the public but are advertised as perfect proxies. People then accept this claim unthinkingly. It's a problem.

Did you just coin that? Looks like previous instances on google were trying to say tourettes syndrome.

I like the phrase, I have been wondering about this problem. Future web crawls are going to contain ever increasing amounts of gpt generated content.

A similar problem is who is going to use stack overflow when an llm can do a better job for simple problems?

Yes I did, though I thought I originally posted it on Twitter or HN long ago.

It’s on the internet somewhere from me at some point previously

That is hilarious, and it utter defeats one of the quality checks on MTurk style tasks: using agreement as a proxy for correctness.

You could potentially introduce an occasional prompt and check the answer against a few LLMs to see if they match.

Good LLM outputs are typically not the same for every identical query. For now that makes AI checkers fairly incompetent (and leads to disastrous teacher results when trying to find the students using AI).

If I ask even just GPT 3.5 something like: "Who was John Adams?" - it'll give me a slightly varied answer pretty much every single time (even if I modify its settings to make it less creative with responses, it'll still usually vary a bit).

Here is a simple API hit on gpt-3.5-turbo-16k-0613

Output 1) John Adams was an American statesman, lawyer, diplomat, and Founding Father who served as the second President of the United States from 1797 to 1801. He was one of the key figures in the American Revolution and played a crucial role in drafting the Declaration of Independence. Adams also served as the first Vice President under George Washington. He was known for his strong advocacy of republicanism and his belief in a strong central government. Adams was a prolific writer and his letters and writings provide valuable insights into the early years of the United States.

Output 2) John Adams was an American statesman, lawyer, diplomat, and Founding Father who served as the second President of the United States from 1797 to 1801. He was one of the key figures in the American Revolution and played a crucial role in drafting the Declaration of Independence. Adams was also a strong advocate for the separation of powers and a strong central government. He was known for his intellect and commitment to public service.

And so on.

What if every output from a prompt was a QR code which led the user to a page with the actual prompt and the output and this is what AI was relegated to do in order to ad some modicum of proof-of-prompt-custodianship...

so the only way one could see the output of the AI was to read the QR code which resulted in the intput & the output?

LLM content is going to be pervasive, to understate it. This stuff is going everywhere. It won't be relegated.

The open source variations in the wild aren't going to necessarily follow any such rules. So even if eg a government wanted to lock it down that way, the mass of AI content spilling over the walls anyway would void the effort.

I doubt the genie is going back into the bottle or can be easily controlled. We'll come up with some basic ways to discern AI content, one of which will be a government regulation (or standards body) stipulating all AI content must/should be labeled as such (with movie like rating gradiants: entirely AI (E-AI), partially AI (P-AI), no AI (N-AI); or something like that). If we're lucky it'll just be a content industry standard that will be developed, without forced government labeling. QR codes or the equivalent could play a role there (QR code next to the content label; end-point data includes where it came from, when), although someone has to host a permanent repository for the end-point data.

… then people would just copy-paste the output after scanning the QR code?

Like, what??

How's would you check that? What if they're using a self hosted LLM?

You think mturk workers have the resources and expertise needed to set up their own GPT?

That doesn't address my question. I didn't ask anything about their resources or expertise.

There's the paper that shows ChatGPT outperforming MTurk Crowdworkers and another one showing GPT-4 going to to toe with experts (and blowing past crowdworkers) who set the benchmarks they were evaluating.

https://arxiv.org/abs/2303.15056

https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro...

If quality is the issue then rest easy lol. The age of state of the art artificial Natural Language Processers being obviously inferior to Human Performance has come and gone.

Out of curiosity, what happens to an LLM if, over time, it's increasingly trained on its own output and/or that of other LLMs?

And has anyone figured out a good way to minimize that when sourcing training data from e.g. Reddit?

I certainly have no idea but interesting to think about! One could argue that humans are trained on their own output so presumably interesting things could arise, especially if it's not a totally closed loop (LLM#1 -> some human curation -> LLM#2).

> humans are trained on their own output

And it seems we're emulating that same cycle. Humans are trained not only on our own output, but also take changes from our environment. So in this sense LLM's environment would be human input.

the problem with the current tech is that the solid part of knowledge isn't separate from the fluid parts. You need to fundamentally understand that human knowledge in many generations was purposely fixed, by books, scholars, etc.

Until that occurs, it's really just a miasma of bullshit.

I read (probably on HN) where someone suggested that pre-LLM data will become the 'low background steel' (pre-Nuclear-age steel with lower levels of internal radiation) of machine learning.

It seems like provenance/traceability of data sources will become important, both for the builders and users of LLMs.

There's a no-longer-very-speculative bit of Sci-Fi waiting here about whole societies "forking" based upon the curation of their training data. Arguably social-media + bots + paid state actors means we're already there.

Isn't "societies forking based upon curation of their training data" practically what culture is?

Now, I agree that having scalable generative AI and high-throughput-high-virality mass media (social networks) brings new unspeakeable horrors like feedback loops to the mix. Interesting times indeed.

can you point to an actual paid state actor. the term seems like mostly bullshit.

Context drift.

https://news.ycombinator.com/item?id=36319076

I believe that performance degrades, as the models end up training on data that is more homogenized and less diverse when compared to original data.

At least according to this paper (I think, FYI I am not a expert) https://arxiv.org/abs/2305.17493

It can reinforce its own biases for one

Have you seen Multiplicity?

"Hey Steve, come on up, I'm spittin on bugs"

Hm, sounds like dog-fooding, but where the 'food' is dog shit. Dog-shitting? Dog-shit-fooding?

Dog breakfasting

dogging

Dogenshittification

The Poopboros.

AI Centipede

Hopefully some of this is democratized away by humans voting on quality of output directly and/or indirectly?

Habsburg AI – a system that is so heavily trained on the outputs of other generative AI's that it becomes an inbred mutant, likely with exaggerated, grotesque features

from https://twitter.com/jathansadowski/status/162524580321127219...

Or even, the Zuckerberg of AI in a meta-sorta-way''

This was pretty clever:

>CaligulAI <-- @Rogntuudju

> CaligulAI

I believe it's spelled "Caligulae" because it's first declension.

/Latin-joke

LOVE it

You know what would be funny ;

."Latin Joke Explainer" -- As explained in Latin Man-Splain-Terms."*

EDIT:

(You mastered my joke, and I appreciate it.)

EDIT AGAIN:

UI think we actually just coined a term ; 'Caligul::AI' -- Malicious AI for its own pleasure.

Lately I have been trying a new LLM called Falcon that was supposedly created in the UAE but it was mostly trained on conversations with ChatGPT... the result is that if you ask Falcon who created it, it will happily say OpenAI. I found that really amusing.

The instruct trained variation, you mean.

You’ve heard it at a concert when the microphone picks up sound from the speakers and a positive feedback loop ensues. A lot of screeching.

I read a research paper which says LLMs don’t actually improve by training on the outputs of other LLMs. They only gain the ability to answer the specific questions included in the LLM output training dataset. Their reasoning capabilities etc. don’t improve.

I’m probably wrong. See the Orca model which came out recently.

Perhaps the source training data will shift to transcripts or subtitles of podcasts, cable television, TV shows, and other sources that haven't (yet) been polluted.

Combined with pre-LLM data sets.

Possibly such data will be used in a different way. If it's embedded in a social network, engagement (votes, retweets,wwillhstecer) will act as a sort of crowd sourced reinforcement learning source instead of being direct part of the training set.

"Model Collapse"

https://venturebeat.com/ai/the-ai-feedback-loop-researchers-...

Training is becoming multi-modal anyway and will need grounding in physical reality so human-generated text will most likely become a smaller fraction of the data.

What will happen is that it will stratify people even more and income inequality will become even more pronounced.

The educated people will learn how to write on their own and the less educated will use LLMs and never rise above it. That will be the McDonald’s workers of the Information Age.

This is the future that I’m preparing my kids for so that they land on top. I’m investing heavily in creativity and writing for them so that they will land in the upper levels of society and not the lower ones that become slaves to AI.

A friend was joking that LLMs explain star trek's obsession with the 20th and 21st century.

Funny because I remember in an AWS conference that the presenter touted Mechanical Turk as AAI (Artificial Artificial Intelligence) because it was a fake AI service done by humans, and it seems that it will become soon AAAI (Artificial Artificial Artificial Intelligence, or A3I).

I had a side project I worked on for 6 months several years ago to do something similar albeit slightly less shady. When I was working on it there was a business on their that was using it to do human transcription of retail store receipts. I think it was a marketing company to figure out what customers were purchasing. My plan was to develop a computer vision algorithm that would do 90% of the work and then just farm out the verification step to humans to increase their productivity.

I got as far as downloading the receipts and attempting basic image processing to try and touch up/reformat the receipts. Training a model to extract data would have been a better approach but I didn't have the skillset or inclination to continue.

Also, the revenue was there for some nice side income (~40k/mo), but not enough to support a company around it. I could have probably used services from an AI/ML startup doing this but it would have eaten into the margins.

There was an article yesterday indicating that feeding the output of LLMs back into them as training data causes the LLM to erase the original content. That effect, combined with people using Mechanical Turk to do pre-processing of content for LLM training, will make things worse.

I can see repositories of pre-2022 textual content becoming valuable. Anything later will have too much circular corruption to be used as training data.

It's this sort of thing that makes me bullish on OpenAI, oddly enough. I keep hearing rumours that it's possible to watermark LLM output with a sort of unique key such that the resulting text still sounds just as fluent, but it can be detected with 100% accuracy given more than a few words and the key. I also hear that OpenAI have mastered this technology.

If that's true then it leads to the possibility of converting first mover advantage to absolute market dominance:

1. Spend lots of money to give a high quality LLM away for free (ChatGPT). Rapidly achieve market dominance by doing so.

2. Watermark its output without making a detector publicly available.

3. Users flood the internet with watermarked text.

4. Now you can generate fresh training data without circular corruption because almost all the LLM output on the internet is yours.

5. Your competitors incorporate LLM output into their training inputs because they can't detect it. Their quality suffers in ways yours don't.

6. Users notice the quality gap and continue to use your model. Competitors can't close it because they have no way to do so.

7. Profit!

This sort of loop happened with Google where using the click stream to do ranking yielded a virtuous cycle that competitors struggled to beat, and then sites using robots.txt to ban non-Googlebots made it worse. If this equivalent actually happens then OpenAI can end up having an absolute monopoly on LLM technology.

This is plausible, but I can put some bounds on the scope of the conspiracy. OpenAI has already released their own AI text classifiers[0], and they are nowhere near good enough to suggest the use of stylometric[1] watermarking. Furthermore, if such a watermark did exist, it would be detectable by anyone with a large enough corpus of known AI text. It probably would also be easier to detect (have lower false classification rates) than the unintentional stylo in ChatGPT output that most AI plagiarism detection is currently identifying.

I could see OpenAI archiving literally every line of text that their models generated, and then doing normal full-text search on their text corpus to remove their own output from it. But there's still one other flaw: the existence of other models.

GPT-3+ are not a monopoly. Google has BERT, Facebook has LLaMA, and the FOSS community has BLOOM, StableLM, and CerebrasGPT. While these models may or may not be GPT-4 quality, they will still be generating text without OpenAI's watermark, meaning that OpenAI won't know to filter for it. So even if they did plan to use watermarking as a way to invisibly scrape human content while polluting other companies' training sets, they wouldn't be able to succeed on such a plan.

[0] https://openai.com/blog/new-ai-classifier-for-indicating-ai-...

[1] Stylometry is the process of fingerprinting word usage to determine authorship. This was used, for example, to unmask "Robert Galbraith" as the pen name of J.K. Rowling, before she made it too easy by just using her mystery novels as a way to rant about trans people and Twitter. More famously, the Unabomber was unmasked as Ted Kaczynski using the same means. Anonymity is a polite fiction and how you write is personally identifying information.

This assumes you can actually watermark the text, which sounds like total bullshit ?

Not really a surprising result. It's a task tailor-made for LLMs and probably somewhat painstaking for a human--especially one that isn't a domain expert which the typical MTurk presumably wouldn't be.

In hindsight it’s not surprising but when you’re paying turkers you expect humans to be doing the work, good to know it’s no longer reliable either

Totally. It's a useful result if only to confirm that people doing tasks that can be handled pretty well by automation will use said automation. Not surprising, but absent a test it's just a supposition based on human behavior. While less problematic, I expect that human transcribers are probably also using ML systems to generate a first pass (though I'm not sure how much that benefits a fast touch typist).

So when these models start learning from more and more AI generated content, can we assume they'll get worse again and eventually make themselves useless?

I'd argue they've never been "good". They're at the level of random blogs on myspace.

The onlybthing that sprang to mind: 'Why so low?'

It's sometimes hard to believe but honest people do exist.

The instructions did not tell the workers they couldn't use other tools. https://github.com/epfl-dlab/GPTurk/blob/main/hit.html

MTurk has had this issue for years. If you tried to us MTurk to do any text or image labeling since at least 2018 you were majority of the time getting output from a poorly functioning ML system.

I am generally not very supportive of restricting AI research via regulation.

However, I would support a mandatory disclaimer indicating that content was created using LLM. Consider it a “truth in sourcing” rule.

A blanket regulation like this would be difficult to manage, because you're dealing with text. It might very well be generated in another jurisdiction that does not have such a rule. Even the service presenting you the text might not be aware that it's generated.

Seems like it could be enforced like GPDR?

I actually don’t have a good idea on how this would be practically enforced. But I think we could start somewhere.

Bullshit generation as a service.

Well, at least this makes it stand out how much of that was going on manually before it was automated.

How long till OpenAI et al. offer a plagiarism detection service against this?

OpenAI are already attempting, they have a free classifier you can try, but results aren't all that great:

"Our classifier is not fully reliable. In our evaluations on a “challenge set” of English texts, our classifier correctly identifies 26% of AI-written text (true positives) as “likely AI-written,” while incorrectly labeling human-written text as AI-written 9% of the time (false positives). Our classifier’s reliability typically improves as the length of the input text increases."

https://openai.com/blog/new-ai-classifier-for-indicating-ai-...

But they don't need a fancy AI classifier, they already have the text that ChatGPT outputs

They'd be absolute fools not to be keeping a copy of everything it's ever said. How much text has it generated, ever? A couple of dozen terabytes? Maybe?

According to https://help.openai.com/en/articles/7730893-data-controls-fa... if you turn off history the conversation will be permanently deleted after 30 days.

Assuming you're actually going to do anything with the results, basically any meaningful number of false positives is a huge problem. If using an LLM (or appearing to have done so) to at least help write a college essay is considered plagiarism, that can easily be grounds for expulsion.

> while incorrectly labeling human-written text as AI-written 9% of the time (false positives).

Utterly useless for teachers. You can't go around accusing 1 in 11 innocent students of plagiarism, and over the course of a semester you'd have virtually every student get falsely accused at least once. They may as well have not released this tool at all.

So far, the results aren’t great.

People claiming to be able to reliably detect LLM generated content have been found to be selling snake oil.

While the general conclusion of this paper seems perfectly plausible, I'm not sure how much weight I can put into the exact numbers they come up with.

They only gather 46 summaries from MTurks, and they train a predictor on some known-good and known-synthetic data, and use that to decide when the crowd workers have submitted LLM-generated text. I'm not totally sold on the idea that the good performance on the validation set is going to translate to this new dataset. In general, I'm just really skeptical of results that say "we can detect generated text with 99% accuracy!".

It would be very interesting to see a study that asks crowd workers afterwards if they used an LLM to complete the task. Obviously you'd be pretty worried about lying, but maybe there's a way to structure the situation so that the workers know their response is anonymous and won't affect whether they get paid.

I wonder how many of those tasks were for training LLMs.

they use mturk in LLM and other generative AI research as operationalized human evaluation

we are doomed to the fake web theory.

Reply on news.ycombinator.com