Signs of undeclared ChatGPT use in papers mounting
retractionwatch.com
retractionwatch.com
But regardless, this is extremely troubling in terms of quality. The fact that "regenerate response" gets left in tells us that even if it's just for grammar correction:
1) The authors aren't even minimally reviewing if the ChatGPT version didn't accidentally change the meaning of the sentences
2) Peer reviewers clearly aren't even reading the paper
3) The journal isn't even reading the paper to introduce a minimal level of quality control
ChatGPT seems to be the smallest problem here. And thankfully, it is revealing much more serious problems about quality checks in academia at all.
The question is, will anything be done about it? Not about ChatGPT, but about the horrifically irresponsible processes that would let these slip through?
Ignore me. :)
-----
I want to pull apart one specific thing you mentioned, essentially:
> It seems totally unclear whether ChatGPT is being used to invent fictional methodology [...], which is extremely unethical
In the context of generating something fictional, why is using ChatGPT unethical?
One reason I'm asking: I have always been a big fan of NaNoWriMo[1] (National Novel Writing Month, every November). This year, I'm interested in exploring how to use ChatGPT -- not necessarily to generate a ton of prose, but to essentially help flesh out some plot ideas that I've had.
Part of the back and forth I've had with it is coming up with some interesting scenarios for my story that I hadn't really thought about and I'm keen on exploring a bit more.
[1] https://en.wikipedia.org/wiki/National_Novel_Writing_Month
The human is the problem in that scenario, not the LLM. The unethical choice happens at the point of deciding to lie about methodology, before LLM is involved.
It may seem like a meaningless distinction, but it’s the difference between LLMs causing trouble and people causing trouble, which is important for the larger debate.
I think one problem with LLMs and the hype around them is that people are placing far too much confidence in them, and therefore misuse them.
That it's even conceivable that people might blame a tool for unethical behaviour is an indicator of this.
The idea that an LLM would be able to correctly generate your methodology for you is another.
I find ChatGPT simultaneously incredible and terrible. It has no concept of what it knows and what it doesn't, so there's nothing I would trust it to do without a human checking the output.
I asked again in a new chat and it made me a really nice one with funny things like a formula showing the amount of force he uses to open doors and fake data proving statistically that he is the best
So apparently someone submitted a paper that was entirely nonsense. This was intentional but not explicit as it was just a stunt. The paper had all the right fancy/technical words, the right structure, likely even images, and apparently a convincing conclusion. But it was all about nothing, nothing substantial at all, nada. Had a peer reviewer actually read the paper, it would have not been published and summarily tossed. But it did go print and oh boy, the journal's board/powers-that-be were pissed. I didn't follow the story much past that point but I think something was supposedly proposed to prevent this.
1. Bad grammar in a paper is incredibly irritating, which is particularly unfair for people who are non-native English speakers. If ChatGPT can help people to convert awkward English into nicer work without changing the ideas or confabulating, that seems like a win. I don't see why people need to "declare" their usage of ChatGPT for these purposes. Is that a standard now?
2. The major issue here is that the authors copied a response and accidentally also brought some button text with it. This is given to imply that people aren't reading ChatGPT's output. But I think this is just as likely to be benign. People do all sorts of silly nonsense in papers (including me), like leaving author comments in final drafts. It's bad and sloppy, but it isn't usually a crime.
3. Many papers on public servers are junky. Many journals and conferences are equally junky. Search for any bad string and you'll find it everywhere, and non-scientists will be scandalized.
4. I like the people at Retraction Watch and think they're doing the Lord's work. However sometimes I worry that immersing yourself in wickedness makes you see wickedness everywhere, and also creates an (unconscious) incentive to sensationalize bad stuff. I hope they're aware of this.
ETA I would urge you not to throw the death penalty at people whose crime may be more akin to shoplifting. This is bad both for the shoplifters and for people who want to fight serious crime. Plagiarism is one of the most serious accusations in science: casually tossing it around like this risks our ability to enforce it against serious offenders.
It's a difference in quality large enough to constitute a difference in kind. Autocorrect nominally takes text in your own words as typed and converts them into your words as intended. Chatgpt used as you describe does not produce text in the author's own words.
As an adult in a scientific field, what actually matters to me is having original scientific ideas conveyed to me in an efficient manner. I'm not really interested in applying arbitrary elementary-school rules on authors, as long as they aren't stealing ideas and text from other people (thus misallocating credit in ways that are harmful to progress.) I think as a community we should proceed from these principles, then work the details out from there.
The elementary school rules, at least the ones I weas taught decades ago, are pretty much “don’t steal ideas and text from other people”. Like, literally, other than for formatting, there were exactly three rules avout plagiarism I was taught on this in elementary school, and only one of them becomes slightly fuzzier outside of that context:
(1) if you use exactly wording from someone else, put it in quotes and cite the source,
(2) if you use an idea from someone else, cite the source but don't ise quotes, and
(3) if an idea comes from three or more independent sources, you don’t need to cite it, it can be treated as “common knowledge”.
Outside of elementary school, yeah, whether something is genuinely common knowledge or should be cited to multiple sources is fuzzier and context dependent, but the other two rules apply pretty directly.
Then the rules you were taught were far more lax than those I learned. According to the rules I was taught, copying lines from your own prior work without attribution would also be plagiarism.
That something concerns matters of little personal interest to you doesn't make it vacuous.
I think the OP's point is that you could be, and should check, as such models sometimes reproduce parts their training set. You should check what you send, or, in other words, take responsibility for the tools you use -- the tools that are bread and butter are somewhat excused, but LLM are not at the moment.
Perhaps a footnote would be added to mention that ChatGPT was used in this way.
Also, if you go to contact the author or view a talk, and suddenly their English is broken and completely different than what they wrote, you start questioning whether they wrote it or paid someone to without disclosing it.
IMO it's worth disclosing for many of the same reasons you disclose funding, ethics approval, author contributions, and conflicts of interest at the end of an article: it's metadata that protects your credibility as an author by showing that you're not doing anything underhanded.
The fact that this slipped by the paper's author, their editor, *and the journal review* is not benign.
I agree with the author here: is no one reading these papers before putting their names behind them? Is the entire journal system entirely fraudulent?
Depends on how much the journal is willing to push the authors (predatory journals don't give a heck, prestigious journals usually care more because they've got a rep to uphold). Reviewers can be anyone from an expert in the field, a competitor who gives you a hard time (usually they're not actively malicious though), to some random prof doing it because the editor couldn't find anyone else, or even a grad student in the lab of a prof (it's a common training exercise).
I've had articles with anywhere from 1-3 reviewers, there is no standard beyond "the editor is a basic crackpottery filter, there is an external reviewer, and a copy editor makes it fit for viewing"
What's interesting is that even when they reply, authors don't seem to respond to the ethics/standards issue of actually using ChatGPT. Well, some do ("I will not do it again" is basically what one of the authors says, which reads like a childish response to me), but many claim it was a "mistake" -- the mistake being... forgetting to remove the boilerplate and "regenerate response" garbage! Uh-oh.
I think an acceptable use of AI would be in translation, basically a better Google Translate. But is this what's happening here?
I have been experimenting with generating powershell scripts using chatgpt, and aside from a few syntax errors or localization, the scripts are a great starting point.
Instead of writing the whole script. I can have ChatGPT write it, then I just have to clean it up, address any errors and localize.
I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. Text-refactor is obviously perfectly ethical, unless you're gunning for the Butlerian Jihad, no worse than the grammar checker in Word.
You and the comment you replied to seem to be making an argument about something neither my own comment nor TFA said.
Nobody is claiming LLMs or ChatGPT aren't useful.
The argument is about using it to write parts of research papers without disclosing this usage. Nobody mentioned using ChatGPT to write code.
Hence my closing line, "[in the context of writing research papers] a valid use of AI would be as a better Google Translate [for non-native English speakers]."
Whether or not they used ChatGPT just as a translation service (an acceptable use, in my opinion), in many cases they didn't disclose this usage even when the rules of submission mandated it.
--
[1] I just checked and now it just says "Regenerate". It must have been "Regenerate response" in the past.
But as with all papers the issue is just a lack of dilligence, be it people straight up using hallucinated sources or the paper bizzarely fawning over every aspect of the project, even things that are objectively rubbish and author knows that when questioned (such as a paragraph praising the high quality of servo motors used when they are blamed for test failure in a later section). Someone taking the time to use it well will use it well, someone lazy will still be lazy.
Exactly. It's a self-solving problem as long as draconian luddites don't outright ban LLMs.
My podunk teachers couldn't fathom the difference between directly citing Wikipedia (just cite the source it references instead) and merely reading it.
Pain.
You also shouldn't cite sources you haven't read yourself.
The work-around is to write it from Wikipedia, but find sources by searching on Google Books. Being able to read a couple pages around each part you’re citing is enough, so their page allowance for restricted works is usually plenty.
Hit your citation count and real-book-source requirement (if they still do that second one) without leaving your chair. Barely more work than writing from Wikipedia and not covering your tracks.
This experience "went full circle" for me, January 2023... when ChatGPT cited SOMETHING I HAD WRITTEN into wikipedia (re: transistor density).
Fixing white balance, or a thousand other old-school photo changes, is very different; the underlying structure of the photo is there and nobody cares about white balance or most of those old corrections, because our eyes and brains are constantly mucking with them anyway. People care about airbrushing or GenAI-"airbrushing" parts of the picture, because the information content we care about is changed.
I was thinking about this a lot recently with the announcement of the Pixel 8. One of the big announced features was "Magic Editor", where you could edit all your photos to automatically put smiles on people who were frowning, remove people altogether like that inconvenient homeless person or garbage truck: https://blog.google/products/pixel/google-pixel-8-pro/
This may be a bit "get off my lawn", but honestly I was disgusted. Why do people feel their everyday family photos need to match some Insta-bullshit ideal? Some of my favorite old family pics are where something "wrong" or unexpected happens.
I don't fault Google for this, and I think it's natural to want to touch up photos when you're taking an important picture. But I feel like we're careening towards a future where we're starting to forget what reality actually is. When TikTok came out with their extremely realistic "makeup filters", I remember seeing a video where a photographer commented that she would take pictures of women who were shocked at how "ugly" they were because they weren't used to seeing "un-Facetuned" photos of themselves.
Sorry for the semi-rant, just feels like a situation where any individual, small technological improvement makes sense, but in totality we end up at a place that is overall worse than where we started.
"Magic Editor in Google Photos is a new experimental editing experience that uses generative AI to help you bring your photos in line with the essence of the moment you were trying to capture."
Then here is the example video they show of a sample photo edit: https://storage.googleapis.com/gweb-uniblog-publish-prod/ori...
After all, why take a picture of your family when your kid is making a silly face when you can instead "fix" him so that you now look like the Stepford-family, JC Penney ad that God intended.
Edit: for further context, I have published a few scientific papers in recent years, and using AI is not all that different from a senior author relying on junior authors. Early on in my career, I created the first draft of most papers, and my more senior co-authors would verify my work and create direct edits or suggestions for me to revise the draft. The final paper was created after a few rounds of this cycle.
There's been an ongoing debate in the Photography community going back as far as photographic editing has existed, regarding when and how acknowledgement of editing has been performed, whether it be literal, physical cutting and pasting of printed media, or a digital equivalent.
Some of the more famous personalities in Photography seem to lean to something like this:
"Minor adjustments like removing dust from the lens, adjusting colors to match the 'true' colors as it was taken (or to make it 'more visually pleasing'), cropping and level correction are fine and expected. The use of things like Sky Replacement [1] or content-aware fills [2] should be attributed or referenced somewhere if they are the focal point of the photograph."
There are others who do much more "artistic" styles of photography [3] where they argue something like:
"No person in their right mind would believe that there isn't some amount of editing going into nearly photograph, so why mention it at all?"
or
"Because editing has been so heavily used, by so many famous photographers for so long, without mentioning editing, it's fine if people don't mention the use of digital (or physical) editing."
or
"Popular services like Instagram have built-in filters that users can use to make themselves look different - should the people who post photos that have those filters on also be pressured into mentioning that their photo was edited? If not, what's the difference?"
There's really not any solid consensus on what is "correct" when it comes to editing attribution. Most tend to agree that doing the bulk of the work "in-camera" (cropping by moving in closer, for example) will make your life easier later on, but, there are some outliers that strongly advocate that any form of editing in post, is "not photography".
Things get even weirder when you get into print. The same photograph printed on different mediums of paper, or with different printers, or different ink, or different DPI settings can come out looking noticeably different even to laymen (imagine a smeared blotchy photo versus a crisp, clear one - there are times where you might want the photo to look somewhat blotchy for aesthetic reasons). In a way, choosing one form of medium over another is a kind of editing, because the end presentation is measurably different from the original. A quick example might be noticing the difference between modern movies that are shot on digital vs traditional film. There is a kind of "feel" to the end product, because of the medium that was used to record and distribute it.
TL;DR The photography community is very much not in agreement to what amount of editing is, or is not permissible, or when it should, or should not be attributed or brought up, or even how that information should be conveyed. Some are fine without any mention whatsoever because of historical reasons. Others want it mentioned under the use of certain tools or criteria. Still other don't consider a photographer to be a "real" photographer at all, and instead should be called a "stills editor" (or something like that) if they use any editing at all, outside of "in-camera work".
[1] https://helpx.adobe.com/photoshop/using/replace-sky.html
[2] https://helpx.adobe.com/photoshop/using/content-aware-fill.h...
[3] https://www.lookslikefilm.com/2019/08/15/artistic-photos/
We know people make up their data. We know peer reviewers sign off on it without actually looking it over. And we know the journals publish it anyway, and make a lot of money off of it.
Why does anybody even care if we've added another layer of fabrication to the mix of something that's already so full of lies?
I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?
Sure, why not? Yes, it's imperfect and BS can get through. But on the whole, it works reasonably well.
This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.
Oh, I agree. However, the peer-review system does mitigate this problem to a useful degree. Without such a system, it would be impossible to have any degree of trust in any paper at all.
> What we're really finding out is that journals are often little better than LLMs as information sources.
It depends on the journal. There are absolutely crap ones out there that need to be ignored. But there are also good ones out there that have earned their reputation. Even there, BS can get through of course -- but to say that such journals are mostly unreliable is seriously overstating the issue.
The idea of peer-reviewed journals is great. The reality is a circle jerk.
And how can one easily determine, while looking at a particular paper, whether it has been replicated? And whether those doing the replication have any undisclosed ties to the original?
At an epistemological level, the idea of a knowledge source like a journal where the information is only deemed reliable if personally verified seems problematic. Why even have it if all of its uncountable claims are indistinguishable from very clever lies, and attempts to quantify the extent of those lies indicate that they are pervasive?
Authors, referees, and journals go through the motions of what lead to genuine scientific advancement in the past, but no actual value is being created at any step in the process for that vast majority of papers. Of course most papers will never be ground breaking but at least in the past there was some value being created in the process.
fix prose, absolubtely. To write?
Well a lot of the more egrerious errors ive seen when trying to get it to write academic writing comes from it not having a full context of what the problem is/whats been done/ what you are trying to do. Ive been turning over the idea in my head that for one to fully give it all the required context one might as well just use that context as the piece of writing in the first place. There is no advantage into stretching the relevant information to 6 paragraphs instead of 2. Not to say it should never be used but trying to use it while being thorough and actually reading its output has usually led me to extending context more and more and eventually just writing the section myself.
A prompt such as "Why are all the dolphins dying?" would be unacceptable.
A prompt such as rewrite "explain that the dolphins are dying because 30 degree rise in ocean temperaturs made their environments unlivable" that reformats the language into a academic style I feel is totally acceptable and efficient if you read what it spits out and agree and edit from there.
I think most other people should not use it for writing, because the hard part is thinking about (A) what you _really_ want to say (B) what your audience cares about (C) mistaken assumptions by you or your audience, which need correction in the text. And I think people using ChatGPT to write will give it a vague idea of (A) and usually nothing about (B) and (C).
But we're fast approaching an era where only the LLMs will read the thing, and only to spit out a summary of it. No human wrote it, no humans will read it.
The biggest journals will of course avoid this fate the longest, maybe even putting it off indefinitely. And some of the most useless journals have been doing worse for a long time. At some point though, doesn't academia have to wake up and realize how pointless all of this is?
It's odd not to see Google or Meta try and make a copyright strike system for written text. Copy and paste LLMs are driving people away from google's ad bucks, filling FB with junk. I guess 'the web' is no longer worth defending, de facto.
The is probably going to be true for other types of documents as well, such as business emails. See this article: https://marketoonist.com/2023/03/ai-written-ai-read.html
Such alarmist tones would sound really silly when applied to other dumb productivity tools.
In the end, the quality of the paper is what should be judged.
If there are very obvious mistakes like a leftover pasted in button text, it calls into question whether the authors (or reviewers) were responsibly verifying the content contributed by a tool that is known to sometimes deliver wildly but subtly incorrect results, and thus the quality of the paper content as a whole.
With so many journals demanding papers submitted in a language not native to the author, use of assistive technology to convert the paper so the underlying information can be made known should be expected.
Is there any evidence that it's primarily used by non-native English speakers for this purpose, though? (edit: just saw the list of authors, probably this is the case, at least in this context).
For what it's worth, I'm also 10,000% on board with that use case. I have definitely stared at a blank piece of paper with writer's block because I couldn't write the introduction for a paper before. If ChatGPT gets somebody past that block, more power to it.
Every paper is required to start with some novel variant of "from the dawn of time humanity has wondered if..." when all you want to say is "We discovered some stuff, here it is." I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence.
Reporting novel scientific findings was never supposed to be an exercise in fluent English authorship mastery.
You make a good point. Maybe the bits that can be automated are the zero-value bits, and therefore the best conclusion would be to get rid of them?
In other words, if it can be automated, it's not real content.
I'd be wary of people writing summaries or conclusions using ChatGPT though, because ChatGPT can write very convincing crap that completely misses the point. Someone who is careless enough to forget removing the "Regenerate response" part of the output may be as careless reviewing whether the conclusion truly says what they meant it to say.
As an aside, I've asked ChatGPT 3.5 to provide interpretations of works of fiction of which I know the most common ones (e.g. "this is a parable about [...]", "this is a warning against authoritarianism", etc), and ChatGPT sometimes comes up with stuff that completely misses the point; but if you aren't familiar with the work (haven't read it except for a summary) you could mistake it for a valid interpretation!
You can use LLMs to create fake science, but also use it expedite writing up real science. I have experimented with feeding in a paper that I've read, I know is relevant, and direct the LLM to extract this point out and connect it to this other point. I then go through and edit. Its faster to rewrite existing text (for me) than to generate it the first time.
We should NOT be demonizing tools that increase our scientists productivity when they use it responsibly, rigorously, and ethicly.
And comments
I predict that by 2030, probably 99.9% of all content and published discoveries will be AI-assisted, and outcompete manual stuff for grants and money. Manual content will be as niche as knitted sweaters on etsy.