ChatGPT: 4000 word University assignment got caught out. Wtf do I do
old.reddit.com
old.reddit.com
> Edit2: I am doing a masters, so this will not be taken lightly. And I recieved 2 grants to study it.
Facts to consider before cheating...
This person does not seem to think things through. Glad to see cheaters put in their place, although I wonder what they would have done if ChatGPT didn't exist.
If you meant the OOP specifically, either they would cheat in some old-fashioned way or not cheat at all. Difficult to tell, especially since we don't know whether it was the first time they cheated.
This can be answered, as until late last year it effectively didn't for the average student.
There were 3 main things they'd do (in my experience) for some kind of take-home assignment. In-room or at-home exam cheating are whole different subjects.
1. Pass notes or answers from year to year. Think shared Dropbox folder between cohorts, with papers and solutions to assignments. Simple fix - don't be lazy, make new graded assignments each year. Actually by doing this, students learn more as they practice the previous papers! Short circuit this by giving them the past assignments openly, and encouraging them to work together on them, then give them a fresh assignment to work on. They'll understand it and be in a good place to do it without cheating.
2. Use essay mills to pay someone to write them an "essay". These sites effectively sell bespoke written essays, marketed with pseudo legitimacy as "study materials". Some come with guarantees of uniqueness and that they won't trip plagiarism scanners as they haven't been given to anyone else. Colleges have access to tools that detect content submitted to other colleges previously, as well as content from the open web, so the essay mill pays other more advanced students or graduates to quickly whip up unique essays for paying students.
Without ChatGPT, I can say with confidence this is what they would do, because it's what they did up until late last year.
You could do stylistic analysis or similar to look for changes in writing style over multiple submissions if something seems "off" - with luck you'll find things like variations of spelling certain words, or specific writing style changes. Or the essay might be overly generic and missing the substance of the specific question topic.
3. Straightforward plagiarism via "copy paste", or "copy, reword a few unusual words, paste". Occasionally you get something more evolved like "copy, paraphrase, paste", or use of a thesaurus to swap words out (often inappropriately) to try avoid binary comparison detection. Fuzzy detection usually spits this though. Sometimes you'll simply get the exact same piece of work submitted by multiple students who have colluded and shared their answer with each other.
Part of writing an essay, or at least the type of essay I'm thinking of, is researching and criticizing other views on the subject, so if you find some material to draw from you might a well incorporate it properly.
The other part is presenting your own thoughts - which I assume is where ChatGPT came into play. I'm not aware of anything that could handle this part before ChatGPT (or other LLMs) existed short of presenting other people's thoughts as your own (also plagiarism).
Those kinds of essays are the ones most likely to find themselves the subjects of tech or non tech enabled cheating. They're also effectively low signal to noise ratio measures, and I'd discourage them.
When someone is presenting their own views, they can still steal someone else's... But there's going to be more breadth in these, and (hopefully) more substance in the essays.
I personally don't consider essays to be useful at all - maybe they're appropriate in certain disciplines, but I'd argue the more concerned a professor or department is about language models, the less intellectual merit there is in the subject, or at least the work being used to assess understanding of the subject.
Unfortunately, few educators focus on really trying to assess understanding and ability to apply understanding to new situations, but that's the kind of reasoning that generative models will struggle with. A very quick way to catch language model users out might be to ask them a bit about the same topic, then in a second paper give them their own initial assignment back and ask them to critically evaluate it now. Good luck critically evaluating hollow, vapid GPT waffle in timed exam conditions!
I went back to grad school 15 years ago. Same story, only it had only gotten worse. I dropped out, in part due to the perceived total lack of institutional integrity.
And more generally, is it okay for institutions to provide commercial institutions copies of the work of students who cannot say no? To what degree should students be in control of what happens to their work (i.e., can they put some kind of license to it, to limit what it is used for)? Should the students be compensated when their papers are used to increase the value (i.e., database) of a third-party company?
In my experience, my students simply deny deny deny, even when I have them dead-to-rights.
I mean, I can understand a professor have a strong suspicion of plagiarism, but proving that a student used ChatGPT seems difficult.
ChatGPT has a very verbose, uncanney valley style to it that likely sounded like nothing the student had organically submitted before, it's not that hard to tell.
I've had 2 students threaten me and my university with lawsuits over accusations of cheating (not plagiarism, fwiw). And I had far more compelling evidence than an uncanny valley of verbosity.
I think it's more likely that this story is bullshit (as someone else in this forum has also speculated) than the likelihood that a professor would be willing to go to the mattresses with only a stylistic argument for plagiarism.
> it's not that hard to tell.
It might not be that difficult to tell, but proof is another matter.
edit: The students were bluffing with their threats of lawsuits, but it still made for two unpleasant semesters.
You read a good article so you wonder which bot made this.
You've heard it here first.
https://www.microsoft.com/en-us/security/blog/2021/07/27/com...
The world, society, is not prepared for a scenario where computers generate human quality copy. This is hugely disruptive and will be used by workers and students to cheat their classes and do their work for them for a while.
It's a disruptive technology.
Same goes for anything that has to do with selling written words. Not a small market precisely.
I wonder if there is a transparent proxy in place logging access to ChatGPT when on a campus network. My employer does this, and makes no effort to hide it. Everything is logged and specific things are captured for evaluation later.
Rather than prove the student used Chat GPT (hard for reasons you point out), isn't this better handled as a suspected cheating scenario (collusion, candidate impersonation etc.)
Many institutions have a regulation on the books to allow for oral examination to be carried out for any course at the discretion of the course leader, for the purpose of evaluating a student for award of a grade.
If you suspected a student had used significant outside help, you'd presumably do the same - have an oral examination, and discuss the work, ask them to discuss the subject matter, and test their understanding of it.
At that point, either the student crumbles and doesn't have any level of understanding beyond "paper thin" (like if they paid an essay mill to write the paper), or they have a good understanding. If the latter, then they meet the learning outcomes of the class and have demonstrated this satisfactorily. And if they have learned the content and show an understanding of it, it's effectively acted as a study aid and actually helped them to learn. They now have shown they know the material, so they have passed the course legitimately.
I suspect in the longer term we'll see attempts to ask questions which can't be so easily answered by generative waffle, which actually should be an improvement on many types of examination. How this works in less scientific disciplines will be more challenging - I imagine they'll struggle to create "un-GPT'able" questions.
But colleges adapted (or at least good professors did) to remote examinations during the pandemic - declare the exam "open book", and ask questions that require robust understanding of topics, such that sitting and reading the book will run you out of time. Give students questions with generated parameters so each student gets a slightly different variant of numerical questions, etc.
The solution to language models generating believable or compelling waffle doesn't need to be technical - I suspect non technical solutions may win out. Whether they scale enough (or a professor cares enough to actually sit down with students for a few minutes) is another question, but maybe they should be doing that in the first place?
This, in my opinion, is the only approach that will work. When I ask cheaters to come in to explain the work, they get cold feet.
In my country, that would require the whole class to be subjected to an oral exam; as our examination regulations require strict equivalency of evaluation methods across all students that take the course during the term (contents of the exam may differ across students but the main method of evaluation, and outcomes to be measured, may not).
All you have to do is fail every paper of ChatGPT quality.
However devils advocate here - I took an essay question verbatim from a humanities assessed exercise, and very quickly (think seconds of effort) created a prompt and got Chat GPT to write text each time it was prompted, until told to stop.
I copied the output (which I'd requested references to be included in), and gave it to the professor who set the question.
They very quickly confirmed the language model essay was considerably better than their human students' essays. It was more coherent and readable, better answered the question set, was more "on topic", and even better structured - it followed a structure that introduced a position, then substantiated the position, gave some mention to alternative viewpoints, and compared and justified the position in relation to them.
The topic wasn't hard, you could easily Google copious essays on the topic, but the output wasn't pulled from the web verbatim (even in parts), and it was better than the actual students' work!
The biggest failing was the references - language models generate plausible and readable text... So it invented lots of very interesting and plausible sounding references, none of which existed. The titles sounded relevant. The authors were appropriate to the subject matter, the journals or publication venues were also appropriate and relevant. It's just that none of them actually existed!
I think it's more likely to have happened, by now, than to not have happened.
If you're writing technical reports that have no english or stylistic weighting in the grading, and you're using it solely as an advanced grammar checker and making sure all the ideas in the content are your own, then it could be acceptable.
And as long as it does not infringe on someone’s copyrighted work, then what is the trouble with using AI to write? I’m sure all scientific papers could be made far more comprehensible in parts with the use of assistive ai tech. It’s like spell check but better
Are AI detection tools able to distinguish this (arguably fair) use of AI from "Please write the whole thing for me from scratch"?
I've noticed that some of them will detect "hollow" text (think profile boosting wannabe thought leader type content from LinkedIn), as being a very high probability of being automatically generated. I don't think the text in question was from a generative model (based on when it was posted), but I can see the stylistic traits in hollow, substance-free human generated text, which detection algorithms try to find.
I think you would be safest to avoid using it like that for now, as the kinds of traits sought include lengths of sentences, peculiarity of words in relation to each other etc. These properties are likely to feed through into the final text when you ask it to redraft.
In saying this, automated detection isn't likely to be reliable enough to take action over (in the longer term). Though I guess from a very blunt personal viewpoint, if someone's own writing can be confused with a generative model output, it probably isn't very insightful work to begin with!
Has anybody else run into that issue?
Person probably cheated to get to where they are, and denied someone else more worthy.