OpenAI shuts down its AI Classifier due to poor accuracy
decrypt.co
decrypt.co
The amount of people in the ecosystem who thinks it's even possible to detect if something is AI written or not when it's just a couple of sentences is staggering high. And somehow, people in power seems to put their faith in some of these tools that guarantee a certain amount of truthfulness when in reality it's impossible they could guarantee that, and act on whatever these "AI vs Human-written" tool tell them to.
So hopefully this can serve as another example that it's simply not possible to detect if a bunch of characters were outputted by an LLM or not.
Edit: this tool is as reliable as a magic 8-ball
When a human is miscategorized as a bot, they could find themselves in front of academic fraud boards, skipped over by recruiters, placed in the spam folder, etc.
To make it more concrete on work I am very familiar with: breast cancer screening. If you had a model that outperformed human radiologists at predicting whether there is pathology confirmed cancer within 1 year, but the accuracy was not 100%, would you want to use that model or not?
I agree that there are places where we shouldn't put AI and that checking whether something is an LLM or not is one of them. However I think the sentence above takes it way too far and breast cancer screening is a pretty clear example of somewhere we should accept AI even if it can sometimes make mistakes.
> When a human is miscategorized as a bot, they could find themselves in front of academic fraud boards, skipped over by recruiters, placed in the spam folder, etc.
Is the problem here the algorithms or how people choose to use them?
There’s a big difference between treating the results of an AI algorithm as infallible, and treating it as just one piece of probabilistic evidence, to be combined with others, to produce a probabilistic conclusion.
“AI detector says AI wrote student’s essay, therefore it must be true, so let’s fail/expel/etc them” vs “AI detector says AI wrote student’s essay, plus I have other independent reasons to suspect that, so I’m going to investigate the matter further”
Two people can buy the same product yet use it in very different ways: some educators take the output of anti-cheating software with a grain of salt, others treat it as infallible gospel.
Neither approach is determined by the product design in itself, rather by the broader business context (sales, marketing, education, training, implementation), and even factors entirely external to the vendor (differences in professional culture among educational institutions/systems).
It can be used for some decision (i.e. not critical ones), but it should NOT be used to accused someone of academic misconduct unless the tool meets a very robust quality standard.
> this tool is as reliable as a magic 8-ball
Citation needed
Nearly everything doesn't give 100% accurate results. Even CPUs have had bugs their calculation. You have to use a suitable tool for a suitable job with the correct context while understanding it's limitation to apply it correctly. Now that is proper engineering. You're partially correctly but you're overstating:
> A tool that gives incorrect and inconsistent results shouldn’t have any part of a decision making process.
That's totally wrong and an overstated position.
A better position is that some tools have such a low accuracy rate that they shouldn't be used for their intended purpose. Now that position I agree with it. I accept that CPUs may give incorrect results due to a cosmic ray event, but I wouldn't accept a CPU that gives the wrong result for 1/100 instructions.
Meanwhile, the leading commercial tools for plagiarism detection often flag properly cited/annotated quotes from sources in your text as plagiarism.
On the other hand, an opaque LLM detector that just prints “that was from an LLM, methinks” (and not e.g. a prompt and a seed that makes ChatGPT print its input) essentially cannot be proven false by an author who hasn’t taken special precautions against being falsely accused, so the bar for sanctioning people based on its output must be much higher (infinitely so as far as I am concerned).
The whole silly concept of an "AI detector" is a subset of an even sillier one: the notion that human creative output is somehow unique and inimitable.
The AI detection tool fails both as it has a low accuracy and could ruin someones reputation and livelihood. If a tool like this helped you pick out what color socks you're wearing, then it's just as good as asking a magic 8-ball if you should wear the green socks.
If you are asking, is this LLM text Human generated, and it says Human (yes), then it is false positive.
If you are asking is this LLM generated text LLM generated, and is says and it says Human (no), then it is a false negative.
So I think the only thing a mythical detector could determine would be LLM, or non-LLM, and let us take it from there. But detectors are bunk; I've had first-hand experience with that.
Sure, it's theoretically possible to add two noisy signals that are uncorrelated and get noise reduction, but is it probable this would be such a case?
It all depends on the properties of the signal and the noise. In photography you can combine multiple noisy images to increase the signal to noise ratio. This works because the signal increases O(N) with the number of images but the noise only increases O(sqrt(N)). The result is that while both signal and noise are increasing, the signal is increasing faster.
I have no idea if this idea could be used for AI detection, but it is possible to combine 2 noisy signals and get better SNR.
You can't detect "truth" from that, but you can often tell (i.e. with better accuracy than chance) whether or not a subject is able to give a confident, uncomplicated yes-or-no to a straightforward question in a situation where they don't have to be particularly nervous (which is why it's not very useful for interrogating a stressed criminal suspect, and should absolutely be inadmissible in court).
But everyone knows that it's not very reliable in almost every circumstance it's used. My point is that while only marginally better than chance, it's still better than chance, unlike the OpenAI's detector, which is significant worse than chance.
You can detect indicators of stress... or hot weather... or stage-fright (admittedly a form of stress)... or too much caffeine... or an underlying (maybe undiagnosed) medical condition, etc. So it does not even necessarily measure "stress".
It's about as useful as the so called "fruit machine" which they used to test for homosexuality[0], in that it is utterly useless while at the same time can be quite ruinous for people. People have been fired over polygraph "fails", and while not admissible in courts, people probably have been fingered for crimes after they failed polygraphs. Also, criminals have gone free after passing polygraphs[1].
>But everyone knows that it's not very reliable in almost every circumstance it's used.
You and I may know that. But a lot of people actually do not. That's why it's still used. Either because people administering those tests think it's "good science", or because those people administering it know that while it's all bullshit the person they are testing might not know that and break down and admit to things. Remember that fake polygraph on the show The Wire, which was just a copier they strapped to the suspect. If I remember correctly that was based upon true events.
A quick google shows e.g. you can hire "polygraphers" to e.g. "test" if your partner was unfaithful, making claims such as: "However, assuming that you have a good polygrapher with a fair amount of experience in working with betrayal trauma, you're going to get results that are at least 90% accurate or better."[2]
The US (and probably a lot of other) government(s) like their polygraphs very much, too[3].
> you can often tell (i.e. with better accuracy than chance) whether or not a subject is able to give a confident, uncomplicated yes-or-no to a straightforward question in a situation where they don't have to be particularly nervous
Uhmm, if somebody sat me down in a room, strapped all kinds of "science" to my body and then asked me questions, I'd be quite nervous regardless of whether I am truthful or not. In fact, I'd be even more nervous knowing it's a polygraph and bullshit, because I cannot know if the person administrating it would know that too.
If that somebody then asked me "Have you ever killed a prostitute?", or "Have you ever colluded with the enemy?", or "Have you ever cheated on your partner?", or "Have you ever stolen from your employer?", for example, my stress would certainly peak despite being able to confidently and truthfully answer "No!" to all of those questions. And I am sure the polygraph would "measure" my "stress".
[0] Yes, that was a real thing too. https://en.wikipedia.org/wiki/Fruit_machine_(homosexuality_t...
[1] E.g. the Green River Killer Gary Ridgway passed a polygraph, so the police turned their resources to another suspect who failed the polygraph. That was in 1984. Ridgway remained free until his arrest in 2001. He killed at least 4 more times after the investigation stopped focusing on him after that "passed" polygraph.
[2] https://www.affairrecovery.com/newsletter/founder/use-abuse-...
[3] https://support.clearancejobs.com/t/the-differences-between-...
Yes, it’s still work, but it’s one step removed from having to think up of the original content.
Nearly impossible across data with a couple hundred characters and dozens to thousands of tokens.
Same goes for representing what it means. If people don't understand statistics or math and such, then show what it means with circles or coins or stuff like that. Point is don't seem ever a good thing for options to get removed, especially if it's for bein cynical and judgin people like they're beneath deservin it. Don't make no sense.
If I have a tool that returns a random number between 0 and 1, indicating confidence that text is AI generated, is that tool good? Is it ethical to release it? I'd say no, it isn't. Removing the option is far better because the tool itself is harmful.
Seems like from what the article we're talkin about says it definitely ain't worse than random by far. Thing you most want to avoid is wrongly labeling humans as AI-written so that seems pretty good. Though it only identified 26% of AI text as "likely AI-written" that's still better than nothing, and better than random. But we don't know or I don't know from the article if that's on the problem cases of less than 1,000 characters or not. It don't say what the *best case* is just what the general cases are.
Anyhow don't seem to me worse than random is the issue here
Might as well remove all comment sections because people suck so assume there's no value having one. Pick any number of things like that. Just ain't a good way to go thinking about anything let alone defending a company for removing it, since the same logic justifies removing your ability to criticize or defend it in the first place. You an AI expert? Assume no, so why we let you talk about it? Or me? People suck so why let you comment? On and on like that.
sounds like AI rather than AI detection to me. :)
That’s a weird kind of extortion, a demand that we placate a subset of the population to the detriment of others. If a conflict came down to people who understand stats versus those blind to it I would put my money on those who understand stats.
The "detector" has extremely little information and the only somewhat reasonable criteria are things like style, where ChatGPT certainly has a particular, but by no means unique writing style. And as it gets better it will (by definition) be better at writing in more varied styles.
That doesn't mean that it can't be distguishable by some other means.
LLMs clearly pass the single sentence test. If you generate far more text than their window, I'm pretty sure they'd clearly fail as they start getting repetitive or losing track of what they've written. In between, it varies depending on how much text you get to look at. A single paragraph is pretty darned hard. A full essay starts becoming something I'm more confident in my assessment.
It's also worth reminding people that LLMs are more than just "ChatGPT in its standard form". As a human trying to do bot detection sometimes, I've noticed some tells in ChatGPT's "standard voice" which almost everyone is still using, but once people graduate from "Write a blog post about $TOPIC related to $LANGUAGE" to "Write a blog post about $TOPIC related to $LANGUAGE in the style of Ernest Hemmingway" in their prompts it's going to become very difficult to tell by style alone.
"All I have to do is modify my virus until the anti-virus doesn't detect it."
>It should only attempt a distinct style if prompted to.
There is no such thing as an indistinct style. Any particular style it could have would be made distinct by it being the style ChatGPT chooses to answer in.
The answers that ChatGPT gives are usually written in a style combining somewhat dry academic prose and the type of writing you might find in a Public Relations statement. ChatGPT sounds very confident in the responses it generates to the queries of users, even if the actual content of the information is quite doubtful. With some attention to detail I believe that it is quite possible for humans to emulate that style, further I believe that the style was designed by the creators of ChatGPT to make the output of the machine learning algorithm seem more trustworthy.
The point being that it's already possible to change ChatGPT's tone significantly. Think of how many people have done "Write a poem but as if <blah famous person> wrote it". The idea that ChatGPT could be reliably detected is kind of silly. It's an interesting problem but not one I'd feel comfortable publishing a tool to solve.
Moreover, the way to deal with AI in this context is not like the way to deal with plagiarism; do not try to detect AI and punish its use.
Instead, assign it's use, and have the students critique the output and find the errors. This both builds skills in using a new technology, and more critically, builds the essential skills of vigilance for errors, and deeper understanding of the material — really helping students strengthen their BS detectors, a critical life skill.
I listened to a podcast with Scott Aaronson that I'd highly recommend [0]. He's a theoretical computer scientist but he was recruited by OpenAI to work on AI safety. He has a very practical view on the matter and is focusing his efforts on leveraging the probabilistic nature of LLMs to provide a digital undetectable watermark. So it nudges certain words to be paired together slightly more than random and you can mathematically derive with some level of certainty whether an output or even a section of an output was generated by the LLM. It's really clever and apparently he has a working prototype in development.
Some work arounds he hasn't figured out yet is asking for an output in language X and then translating it into language Y. But those may still be eventually figured out.
I think watermarking would be a big step forward to practical AI safety and ideally this method would be adopted by all major LLMs.
That part starts around 1 hour 25 min in.
> Scott Aaronson: Exactly. In fact, we have a pseudorandom function that maps the N-gram to, let’s say, a real number from zero to one. Let’s say we call that real number ri for each possible choice i of the next token. And then let’s say that GPT has told us that the ith token should be chosen with probability pi.
https://axrp.net/episode/2023/04/11/episode-20-reform-ai-ali...
I personally would want to live in Aaronson's world, and not the world where a centralized authority controls the definition of reality.
However, as soon a detection tool becomes publicly available (or even just the knowledge that watermarking has been implemented internally), a simple enough garbling LLM would pop up that would only need to be smart enough to change words and phrasing here and there.
Of course these garbling LLMs could have a watermark of their own... So it might turn out to be a kind of cat-and-mouse game but with strong bias towards the mouse, as FOSS versions of garblers would be created or people would actually do some work manually, and make the changes by hand.
The water marking techniques also can not work after some level of sophisticated rewriting. There simply will be no data encoded in the probabilities of the words.
You don't see the writing on the wall? OK, here is a big hint: it might make a huge difference from a legal perspective whether some "photo" showing child sexual abuse (CSA) was generated using a camera and a real, physical child, or by some AI image generator.
A world where I can't tell if something is made by human or by machine is a world that has been drained of something important to me. It would reduce the appeal of all art for me and render the world a bit less meaningful.
The upshot of which is, the useful writing assignments I used to give as homework will either have to be done in class (wasting valuable class time) or given up altogether (wasting valuable learning experiences).
And as a teacher who really WANTS them to learn and to get that feeling, "Hey, I can actually do this!", it's depressing to think of the one who do cheat themselves. Oh well...
I think you could pick up right quick on who understood what they wrote, and who didn't.
They once said I should join Line then we can all talk, then I asked if it's possible to talk in groups of 200+ and their eyes got really big.
If your students want to betray themselves of the possible learning opportunities of attempting to formulate the sentences by themselves in English, it is their problem.
The same holds in mathematics (degree course): of course, in the first semesters, you can use a computer algebra system like Maple or Mathematica for computing the integrals on your exercise sheets, but you will betray yourself of the practice of computing integrals that these exercise sheets are supposed to teach you.
These models are clearly not good enough for decision-making, but still might tell an interesting story.
Here's an easily testable exercise: get a load of news from somewhere like newsapi.ai, run it through an open model and there should be a clear discontinuity around ChatGPT launch.
We can assume false positives and false negatives, but with a fat wadge of data we should still be able to discern trends.
Certainly couldn't accuse a student of cheating with it, but maybe spot content farms.
I saw that this report came out today which frankly is baffling: https://gpai.ai/projects/responsible-ai/social-media-governa... (Foundation AI Models Need Detection Mechanisms as a Condition of Release [pdf])
So such a model is doomed from the start, unless its parameters are a closely-guarded secret (and never leaked). Then it means it's foolable by those with access and nobody else. Which means there's a huge incentive for adversaries to make their own, etc. etc. until it's just a big arms race.
It's clear the actual answer needs to be: we need better automated tools to detect quality content, whatever that might mean, whether written by a human or an AI. That would be a godsend. And if it turned into an arms race, the arms we're racing each other to build are just higher-quality content.
could you contextualize your use of the word "easily" here?
I feel like "easily" might mean "with infinite funds and frictionless spherical developers."
LLMs are newer than GANs afaik, it just so happens GANs are a good fit here, not that one is "smarter" or "dumber".
even if an LLM can give an amazing and correct answer 7/10 times, it still takes a human expert to cherrypick which 7 answers are amazing and which are just convincingly-assembled bs.
I listened to an interview with the StabilityAI founder / ceo the other day. He said we should think about LLMs like having a bunch of clever grad students / interns floating around that we can freely offload tasks to. They aren’t experts, but they’re very diligent. The question is, how can we effectively make use of them? People who succeed at this will be much more productive.
> Write a training plan for a series of lessons to teach someone modern deep learning. The training plan should last for approximately 3 months of lessons.
> The lesson plan is for a single student with a strong background in programming (systems programming, algorithms and web). But the student has little knowledge of python. And university level mathematics knowledge but relatively weak skills in linear algebra, probability and statistics.
> By the end of the training process, the student should know modern deep learning methods and techniques and be able to modify, implement and deploy AI based systems.
> Think through your answer. Start by listing out learning objectives, then write a teaching plan to meet those learning objectives.
The response from chatgpt was super long! It gave me recommendations for what to study each week for the next 3 months. I've started going through the material it recommended. For the first 2 weeks, my goal is to learn the basics of python, and learn some linear algebra, and probability and statistics. Then its just a case of finding appropriate material online. I'm watching a lecture series on youtube teaching matrix mathematics now.
[1] https://chat.openai.com/share/d6966012-0d96-4511-b96e-086b80...
It's like asking a 747 to be made into a dog.
It's completely nonsensical to me.
(that said, "may eventually be possible" is so weak a claim it's already meaningless. Quantum fluctuations may eventually turn me into a potato but it's not keeping me up at night)
An analogous example: my local pizza delivery (where I worked) would shut the box with a safety sticker, to avoid tampering / dipping by the delivery boys. Now, sometimes they would forget to do this for various logistical reasons. Every one of the non-stickered ones started getting returned as customers worried a pepperoni stolen. They stopped doing it shortly after.
Kinda like if they forgot to put the security seal on your aspirin, I'm not going to take them all off because someone forgot to run production with all the bottles sealed.
This is the brown m&m principle in effect.
The kind of people that can't get a job at a pizza place.
Personally, I never order delivery through these services. The incentives are all wrong. Not to mention the costs are super high: restaurants don't make any money, I pay out the @$$, and the drivers are given sub-minimum-wage pay after taking on the risks of delivery driving.
The tool in question was used for AI text detection not generation.
Absolutely not.
Of course the smart student will easily figure out a way to stream the GPT output into Google Docs, perhaps jumping around to make "edits".
A clever and unethical student is pretty much undetectable no mater what roadblocks you put in their way. This just stops the not clever ones. :)
Which sucks, because take-home projects are evaluating a different skill set, and some people thrive on one vs the other. But it is what it is.
You could program a robot to re-type the ChatGPT output into a different word processor and feed it parameters to make the duration between keystrokes and backspaces fluctuate over time. You could even have it stop, come back later, copy and paste sections and re-organize as it moves through and end up with the final essay from ChatGPT.
In the same vein that letting students write calculator programs to do the quadratic formula for them during a test is actually a pretty good way to get them to learn the quadratic formula.
It's not an exact equivalent to the intent of an essay today - from an education perspective - but it's not a complete miss either.
Why not? Record a bunch of humans writing, train model, release. That's orders of magnitude simpler than to come up with the right text to begin with.
No need to complicate it that much. Just start off writing an essay normally, and then paste in the GPT output normally. A teacher probably isn't going to check any of the revision history, especially if there's more than 30 students to go through.
Yes, anybody can write an agent to meander about typing the chatgpt generated text into Google docs. Yes, Google could judge how likely it's that a document was typed by a human, but they won't for the same reasons openAI just cancelled this.
Somebody (maybe reacting to this news, maybe reading this thread) will write such an editor or evaluator. Another solution is screen recording as you write. Another (the best one, and the hardest one for educators) is to not request or grade things a robot can write better than most humans.
The education bubble is about to implode - it will probably be one of the first industries killed by AI.
This was my conclusion as well testing the image detectors.
Current automated detection isn’t very reliable. I tried out Optic’s AI or Not , which boasts 95% accuracy, on a small sample of my own images. It correctly labeled those with AI content as AI generated, but it also labeled about 50% of my own stock photo composites I tried as AI generated. If generative AI was not a moving target I would be optimistic such tools could advance and become highly reliable. However, that is not the case and I have doubts this will ever be a reliable solution.
from my article on AI art - https://www.mindprison.cc/p/ai-art-challenges-meaning-in-a-w...
Could it be that a large proportion of the source stock photos were actually AI generated?
This is really painful, because for some of my work I need high quality images suitable for print. Now I can't just look at the thumbnail and say "this will work". I now have to examine it taking more of my time.
Starts talking like Shakespeare
its as Sisyphean a task as AI detection.
Possibly (who am I kidding. *PROBABLY*!) will use chatGPT to help them design the method :)
Cryptographic signing means "I wrote this" or "I created this". Sure you could sign an AI generated image as yourself. But you could not sign an image as being created by Getty or NYT
A chain of trust is one way to solve this problem. Chains of trust aren't perfect, but they can work.
But if you're going to build a chain of trust that relies on humans to certify they used a non-tampered-with crypto camera, why not just let them use plain ol cameras. Adding cryptosigning hardware just adds a false sense of security that grifter salespeople will lie and say is 'impossible to break', and non-technical decision makers wont understand the threat model.
From my understanding, this is a fools play in the long run, but there are current Ai Classifier Detectors that can successfully detect ChatGPT and other models (Originality.ai being a big one) on longish content.
Their process is fairly simple, they create a classification model after generating tons of examples from all the major models (ChatGPT, GPT4, Laama, etc).
One obvious downside to their strategy is the implementation of Finetuning and how that changes the stylistic output. This same 'heavy hitter' has successfully bypassed Originalities detector using his specified finetuning method (which he said took months of testing and thousands of dollars).
The current state of Google is a disaster, everything is 100 paragraphs per article, the answer you are looking for buried half way in to make sure you spend more time and scroll to appease the algorithm.
I cannot wait for them to sink all these spam websites.
If we accept this ...
The challenge I am foreseeing is this:
We are only at the very beginning of the AI revolution -- and if LLMs need to get more sophisticated and powerful in future they will need good-quality human-generated / curated training data at a scale that is likely impossible to do manual curation/cleansing/quality-checks on.
And there is no doubt that evey medium is going to get bombarded and spammed with AI-generated content in coming years.
How then, are we going to filter the data -- to separate the real data from AI generated noise -- to train future LLMs on -- and really push them to their potential.
This problem has been bugging me for a while and I commented here previously as well, tentatively calling it 'Data Pollution' for the lack of a better word.
Curious to hear other perspectives on this.
¯\_(ツ)_/¯ try paper I guess. Time to brush up on our OCR.
But you know who has more real-world data on typing style? Google, Microsoft, Meta, and everyone else who runs SaaS docs, emails, or messaging. I imagine a lot of students write their essays on Google Docs, Word, or the like, and submit them as attachments or copy-paste into a textbox.
Maybe a better term would be Superior Intelligence (SI). I sure as hell would not be able to pass any legal or medical exams without dedicating the next decade or so to getting there. Nor do I have any interest in doing so. But chat gpt 4 is apparently able to wow its peers. Does that pass the Turing test because it's too smart or too stupid? Most of humanity would fail that test.
So assuming all that to be true, how can the likes of Turnitin claim to be an authority for AI writing detection. When I graduated a few years back, they used to offer plag check only.
Pretty easy - they lie to people.
If the first does a good job, the second fails. And vice versa.
(On the other hand, maybe there is a lot of money to be made selling both, to different groups?)
Only people using it deceptively would be affected. No idea what portion of ChatGPT's users that is, would be very interested to know.
It wouldn’t beat determined users but it would at least catch the unaware.
Also the cost of storing every paragraph hash might eventually add up even if at the moment it would be negligable compared to the generation cost.
Did someone say Bloom Filter??
For educators looking at evaluating students, essays and the like - we possibly need different ways of evaluation rather than on written asynchronous content for communicating concepts and ideas.
For civics, I would say yes.
Imagine you were talking to an online group about a design project for a local neighborhood. Based on the plurality of voices it seemed like mist people wanted a brown and orange design. But later when you talk to actual people in real life, you could only find a few that actually wanted that.
Virtual beings are a great addition to the bot nets that generate false consensus.
https://www.reuters.com/technology/openais-sam-altman-launch...
Now the topic isnt about anything millennial or Zelda related, but I'd think that the language model would select sentence and paragraph phrasing differently.
Maybe I need to switch to the API.
First, it tends to print a five-paragraph essay, with an introduction, three main points, and a conclusion.
Second, it signposts really well. Each of the body paragraphs is marked with either a bullet point or a number or something else that says "I'm starting a new point."
Third, it always reads like a WikiHow article. There's never any subtle humour or self-deprecation or ironic understatement. It's very straightforward, like an infographic.
It's definitely easy to recognize a ChatGPT response to a simple prompt if the author hasn't taken any measures to disguise it. The conclusion usually has a generic reminder that your mileage may vary and that you should always be careful.
If so, nice meta-commentary.
I think this upcoming school year is going to be a wakeup call for many educators. ChatGPT with GPT-4 is already capable of getting mostly A's on Harvard essay assignments - the best analysis I have seen is this one:
https://www.slowboring.com/p/chatgpt-goes-to-harvard
I'm not sure what instructors will do. Detecting AI-written essays seems technologically intractable, without cooperation from the AI providers, who don't seem too eager to prioritize watermarking functionality when there is so much competition. In the short term, it will probably just be fairly easy to cheat and get a good grade in this sort of class.
Besides, even if they did win, they would still lose by shooting their own foot.
It is important humans learn to express themselves in writing. The only way I think this will happen is if kids do their writing at school supervised.
In fact it doesn't take much text to distinguish between two human beings. The humanly-obvious version is that someone that habitually speaks in one dialect and someone else in another must be separate, but even without such obvious tells humans separate themselves into characterizeable subsets of this space fairly quickly.
I'm skeptical about generalized AI versus human detection in the face of the fact that it is adversarial. But a constant, unmoving target of some specific AI in some particular mode would definitely be detectable; e.g., "ChatGPT's current default voice" would certainly be detectable, "ChatGPT when instructed to sound like Ernest Hemmingway" would be detectable, etc. I just question whether ChatGPT in general can be characterized.
In OpenAI's case, its writing style usually comes from OpenAI's in-house dataset they used for RLHF. This is what gives it the ability to chat and respond with its signature (perhaps overly formal and apologetic) tone.
Although it can be used to write in other styles, sometimes it will refuse to because of this.
Not the educators fault tough, more like the system is bad.
My point is that given knowledge is mostly free and available, the system should teach the students to think rather than using tools or remembering facts
bigger text e.g. reports, thesis etc are probably easier & cheaper to verify by humans, with help of A.I. tools (ref checking, searching...)
OpenAI can internally keep a "hash" or a "signature" of every output it ever generated.
Given a piece of text, they should then be able to trace back to either a specific session (or a set of sessions) through which this text was generated in.
Depending on the hit rate and the hashing methods used, they may be able to indicate the likelihood of a piece of text being generated by AI.
Then you have database costs of storing all that data forever.
Moreso, it's only for openAI, I don't think it will be too long before other gpt4 level models are around and won't give two shits about catering to the AI identification police.
That depends on how they hash the data, right? They can use various types of Perceptual Hashing [1] techniques which wouldn't be susceptible to a single-character change.
[1] https://en.wikipedia.org/wiki/Perceptual_hashing
> Then you have database costs of storing all that data forever.
A database of all textual content generated by people? That sounds like a gold mine, not a liability. But as I've mentioned earlier, they don't need to keep the raw data (a perceptual hash is enough).
> won't give two shits about catering to the AI identification police
I'm sure there will be customers willing to pay for access to these checks, even if they're only limited to OpenAI's product (universities and schools - for plagiarism detection, government agencies, intelligence agencies, police, etc).
Here's a decent paper on it.
It covers private watermarking (you can't detect it exists without a key), resistance to modifications, etc. Essentially you wouldn't know it was there and you can't make simple modifications to fool it.
OpenAI could already be doing this, and they could be watermarking with your account ID if they wanted to.
The current best countermeasure is likely paraphrasing attacks https://arxiv.org/pdf/2303.11156.pdf
I suppose hosted solutions like ChatGPT could offer an API where you copy some text in, and it searches its history of generated content to see if anything matches.
> bUt aCtuAlLy...
It's not like I don't know the bajillion limitations here. There are many audiences for detection. All of them are XY Problems. And the people asking for this stuff don't participate on Hacker News aka Unpopular Opinions Technology Edition.
There will probably be a lot of "services" that "just" "tell you" if "it" is "written by an AI."
Pretty common steganographic technique, really.
Additionally, if I copy-paste text like this are the invisible characters preserved? Are there a bunch of extra spaces somewhere?
IMHO it isn't a feasible way of watermarking text though - as someone would promptly come up with a website that undid such substitutions.
It doesn't matter since there's no one-pass solution to counterfeiting.
You have the right of it-- the best you can hope for is adding more complexity to the product, which adds steps to their workflow and increases the chances of the counterfeiter overlooking any particular detail that you know to look for.
Watermarking needs to be subtle enough to be unnoticeable to opposing parties, yet distinctive enough to be detectable.
So, this is an arms race especially because detecting it and altering it based on the watermark is also fun :)
This seems like a total non-starter. That can only negatively impact the answers. A solution needs to be totally decoupled from answer quality.
Their improved method instead only replaces tokens when there's many good choices available, and skips replacing tokens when there are few good choices. "The quick brown fox jumps over the lazy dog" - "The quick brown" is not replaceable because it would severely harm the quality.
Essentially it's only replacing tokens where it won't harm the performance.
It's worth noting that any watermarking will likely harm the quality to some degree - but it can be minimized to the point of being viable.
This would not impact output quality much, but it would only work for longish outputs. And the token probability "key" could probsbly be reverse engineered with enough output.