Scientific sleuths spot dishonest ChatGPT use in papers
nature.com
nature.com
For this service of copy-pasting text into a digital PDF they charge their crazy premium. This is amazing. In a sane world this would finally destroy their reputations so we could replace them with a reasonable solution for evaluating researchers.
This is only news to people outside of academia. Anyone who has ever submitted a paper has gotten comments that can be attributed to the reviewer either being braindead or the reviewer not actually having read the paper. Since it's safe to assume said reviewer actually tied their shoes and put on their underwear the morning they reviewed the paper, it's also safe to assume it's the latter and not the former.
It's funny people are raising their pitch-forks about the journals. I guess it's not widely known/understood that the journals aren't reviewing anything ever. So when I say reviewers I'm talking about PhDs in academia that do this work as "community service". So if peer review is fundamental to the scientific method then your tax dollars are being wasted but the bulk of the waste is on worthless, lazy, BS spouting academics not journal admins.
> and practically all of academia just went along because there were no other options.
Arxiv has been around for a long time now. Do you really think "the academy" couldn't have come to some consensus on how to allocate review duties for papers that are uploaded/stored/submitted there? The fact that they haven't strongly indicates that lazy ass peer review as orchestrated by the journals is to the benefit of the academy.
Journal (and conference) reviewers are completely anonymous and there's absolutely no appeal process (rebuttal phase is BS). Does that sound like a democratic/fair/auditable/meritocracatic setup to you? Sounds like review by inscrutable tribunal to me. So if the process were open/community driven (ie democratic) then people wouldn't be able to hide from the shitty job they do.
No doubt - past two papers I've submitted I can basically deanonymize my reviewers based on the other papers/projects they insist my work is inferior to.
Correct me if I'm wrong, but isn't peer review usually volunteer work? With the job of the (usually paid) editors to select reviewers with a track record of basic competence.
> Do you really think "the academy" couldn't have come to some consensus on how to allocate review duties for papers that are uploaded/stored/submitted there?
Actually, there are several proposals! One such system is https://openreview.net/about – it's not the first I've seen, but it's the most recent. (I first learned about these things from a fictional proposal in a Pokémon fanwork: it's far from the realm of niche.)
> Journal (and conference) reviewers are completely anonymous
To reduce the incentives for nepotistic anti-scientific behaviour. The system's flawed, sure, but it works how it does for a reason.
You need to ask yourself how that track record is constituted ie what metrics decide basic competence. Hint: it's not anything that resembles an objective measure of ability or diligence or literally anything. Another hint: it's *extremely* common for PIs to pawn review duties off on grad students - I review papers for my advisor all the time and I have been since my first year.
> To reduce the incentives for nepotistic anti-scientific behaviour.
This is the thin veil that everyone hides behind to cover up the greater harm of inscrutability^1; everyone knows what everyone else is working on in any field. At this point in my career (I'm towards the end of my PhD) I could tell you which group wrote the paper (if it's in my area) just by reading the abstract.
And if you're thinking to yourself well what about new entrants into the field? There's no such thing - you do not learn XYZ hyper niche thing without being part of a long lineage of forebears doing XYZ thing. And if it does so somehow happen, like a student is doing something completely different from their advisor and so they're not trying to publish in familiar venues, well guess what they get face looooong odds getting their work accepted (ask me how I know :)
^1 Like come to think of it, recently I've been reading that the conservative judges on SCOTUS think proceedings should be closed. And you could make some sort of similar nepotism case about that, or some other conflict of interest type thing (in favor of closing the proceedings) but we all recognize that fundamental to a free and fair process is that it's open and auditable.
Unless the reviews are submitted in the grad students' names, that sounds like academic misconduct. Can you “anonymously” report this without too much backlash? (I assume the answer is no: you clearly know it's wrong, and haven't done anything about it yet.)
> This is the thin veil that everyone hides behind
There are many places in society where we do this. The thin veil of politeness, lies that everyone knows are lies – like asking "How are you?" without actually caring how the other person is, and answering "Quite well, thank you." when you're neither okay nor thankful.
It makes no sense, but people would behave worse without the illusion. I know not whether the benefits of transparency would outweigh that.
You expect a first year junior grad student (e.g., me before i was inured to it) to blow the whistle on their senior, well respected, famous, advisor? Is this a joke? You think change like this happens bottoms up from the indentured servant^1 class? lol.
> It makes no sense, but people would behave worse without the illusion.
wut. we've taken a large step off the sane road here. democratic/fair/equitable processes are open. full stop. if you want to debate me on this you're gonna have to whip out a whole lot of philosophy, history, data, etc.
^1 do you know how many phd students in the US are foreign nationals on student visas? if you had to pick between majority and minority which one would you pick? hint: you're wrong it's the other one.
> democratic/fair/equitable processes are open. full stop.
Secret ballots. Double-blind medical trials. The sealing of juvenile court records.
It's not as clear cut as you make it out to be: what might make sense in one academic field (e.g. applied particle physics) might be total nonsense in another (e.g. partial differential equations).
in the first and the third you're conflating the processes of the governing body with the experiences of the governed - the governed need to be protected from the governing body, not the other way around - that is the whole point of demanding that the governing body operate in the open. like bro politicians (as politicians) have no right to privacy, but we as individual citizens, governed by them do. how is it possible that i need to explain this to you?
the second isn't a democratic process of any kind.
> It's not as clear cut as you make it out to be: what might make sense in one academic field (e.g. applied particle physics) might be total nonsense in another (e.g. partial differential equations).
oh but it is and you'd have to be as obtuse as my reviewers to argue differently. like bro there isn't a scholar/thinker/activist on this subject anywhere in the world that agrees with you (politicians definitely agreew with you though).
> like bro there isn't a scholar/thinker/activist on this subject anywhere in the world that agrees with you
Not a 'bro'; also https://clauswilke.com/blog/2014/10/12/in-defense-of-anonymo... https://doi.org/10.1080/01411920802044438 https://doi.org/10.1038/249601a0 . I pretty much agree exactly with Claus O. Wilke, and haven't read beyond the abstract of the other two, but these were all on the first page of the first web search query I tried.
It's not well-documented, but even OpenReview permits anonymous reviews: see https://docs.openreview.net/reports/conferences/openreview-n...
> After the notification deadline, accepted papers were made public and open for non-anonymous public commenting, along with their anonymous reviews, meta-reviews, and author responses.
Your (incorrect) perception of an overwhelming consensus probably isn't why you hold the position you do. I'd like to hear your actual justification, if you would be happy to share it.
> Your (incorrect) perception of an overwhelming consensus probably
Did you know that the word consensus does not include the word unanimous? Just like there is a consensus on climate change despite some exceptional crank scientists there is a consensus on democratic processes being open - again i will emphasize that i never claimed there was consensus about whether peer review should be open but about whether government should be (and my own claims about peer review are informed by that consensus).
> Not a 'bro';
One of my favorite things is when someone is either so pretentious or so clueless as to reject a (gender neutral) colloquialism.
Yes. Did you remember that you wrote this?
> oh but it is and you'd have to be as obtuse as my reviewers to argue differently. like bro there isn't a scholar/thinker/activist on this subject anywhere in the world that agrees with you (politicians definitely agreew with you though).
If you would like to share the reasons for your opinion about academic peer review, feel free.
> One of my favorite things is when someone is either so pretentious or so clueless as to reject a (gender neutral) colloquialism.
Huh, so it is. https://languagelog.ldc.upenn.edu/nll/?p=48576 Mark me down for 'clueless'. I'd seen that with “girlie” and “guy” (perhaps coincidentally, both of which match historical uses of the words), but not “bro”. It's not listed in Wiktionary (https://en.wiktionary.org/wiki/bro#English), but it looks WT:ATTEST-able: do you have a definition that could go there?
This is so extremely common that I don't believe that there is anything to report. I reviewed dozens of papers under my PI's name throughout my entire PhD. Now that I have completed my degree, I never want to review another paper again.
If it's that dishonest, I don't want anything to do with it.
Thank you both for being honest.
my hypothesis on the value of a PhD: if you are the kind of person that can finish an honest PhD i.e., real work that's not necesssarily novel (I don't give a flying fuck about "novelty") but actually requires you to stretch hard to achieve, then you don't need one and it will cost you years of productive/rewarding/lucrative work in industry (and 100% academia is not for you).
i don't know what the "on the otherhand" is though - I really have no idea how anyone comes away from this process thinking "hmm yes I want more of this pile of bs" so I have no idea what kind of psychopaths go into academia (for which the PhD is indeed a prerequisite).
Consider another unethical practice: sexual harassment, in many industries, was once so extremely common that there was nothing to report. Everyone in a position to do something about it knew it happened, yet it continued. People reported it anyway. Tanked some of the reporters' careers, but the practice is way less accepted now.
You've completed your PhD, so you're at less risk of backlash. Are you yet in a position where you can blow the whistle on what happened to you?
Neural necrosis is rarely uniform across the entire organ. Adjust assumptions accordingly.
I don't know, that's pretty questionable, especially since the plague.
We all saw this live with LK-99. Publication isn't the end, it's the beginning. Replication is literally all that matters.
The main services they supply are basically document hosting and search functionality. Both at extortionate rates.
Of course, that isn't always the case and it certainly isn't the case when writing for some of the larger editorials, but it happens a lot. Once you establish a name for yourself, you can get away with a lot. For example, submit your first draft even though it can probably be improved upon if you let it sit.
But the person you're sending it to doesn't know that. They need the story, they know your resume, and they don't want to sit there and go through your submission examining every single word.
Tying professors and researchers careers to the volume of papers they churn out is silly. The result is the antithesis of quality.
Public money is often driving most research but then the results are thrown into an antiquated pay-walled for-profit journal system.
The entire peer review system is rife with issues, from like low-level corruption (like the 'blind' reviewer asking for references in exchange for publishing) to no one even reading the paper.
When some major technology like ChatGPT comes along it's usually a good time to get back to basics while strengthening the foundation through modernization.
It's like Goodheart's law applied to an entire swath of indicators for an entire sector of society.
It's interesting but I think Goodheart's law sort of implies that the metric being gamed is being gamed because there isn't an alternative? Sometimes that might be because what's trying to be measured is difficult to measure, but one possibility is that it's difficult to measure because it doesn't really exist as it's assumed to be.
I'm not trying to say academics are idiots or something like that, if anything the opposite. People in general are just a lot smarter than — or at least don't differ as much — our current dominant schema for scientific contributions recognize.
This normally happened when I context switched from dialog driven storytelling to long blocks of narrative storytelling. So, people should take each problem as a unique problem and not a systemic problem rampant through the system. Also AI check tools are different with differnt thresholds.
To me it feels like AI is part boogeyman too, "What are we going to do with this?! It's clearly our Y2K-borg-Napster-assimilation-with-VHS-and-CDs moment."
I will close by saying, if you are not augmenting your life with AI, you're only hurting yourself. I can guarantee you that your competition is doing this, (both as individuals and for their teams using Midjourney, Grammarly, ChatGPT, etc). I should also qualify that I don't take the hardline ethical stance on AI things (clearly). Some people do. I don't know what do with hardliners, it's damn hard to have a conversation these days.
In other words, the reverse ChatGPT that tries to identify papers written by AI. Is it really sure that the text is plagiarized or is it simply "plausibly" written by AI. That is if something like this goes go to court and the judge asks serious questions... will these AI detectors be laughed at?
Not sure what is more astonishing here: The fact that copypaste makes it into a peer reviewed journal, or that the use of ctrl+f is being called "detective work"
Excel has caught lots of financial crimes. No reason to hate on old text search.
“We can probably go deeper”
Into
“It is postulated that further inquiry may elucidate abstruse intricacies on the topic”
From the article:
Cabanac noticed that some of the equations in the paper didn’t make sense, but the giveaway was above a table: ‘Please note that as an AI language model, I am unable to generate specific tables or conduct tests …’
Declaring usage is one thing, but I think declaring non-usage might be the more valuable thing in the end.
Personally, I love writing - gets me in the flow. I'm weird that way. But I realise lots of people struggle in the early stages of drafting something. If they have a tool that helps them, that's cool. It's just easier to spot mistakes in something someone (or something) else has written than to avoid typing them manually, is all.
I was wondering if I should declare that I did _not_ use any text analysis or generation tools in analyses I produce for clients. I'd wager people declaring that are less likely to lie about it than those using such tools are to not mention it. And someone reading that might find it helpful knowing that they can go over it with less (obviously still a healthy amount of) scrutiny. Declaring it proactively vs requiring disclosure seems like the reverse of the effect opt-out vs opt-in has on user behaviour.
My answer was, by the way, yes. This particular document was in German, but roughly translated I wrote in the first section (where I clarified how I understood the assignment and what goals I set myself):
"Aside from automated spell checking, the analysis behind this document as well as the act of drafting and writing it did not involve - at any stage - tools for automated text analysis or text generation."
I found my peace with this now, but still curious if (and how!) others do this.
Somewhat comparable is that in some contexts journalists and industry analysts, for example, make a point of disclosing any client relationships or investments with companies they're writing about. It's also common to disclose if you were given a product or trip you wrote about.
If I feel at a disadvantage (as in, other people produce better work with LLMs), I'll just adopt them too. But I'm really about quality over quantity, and for my kind of work that's not looking like it's becoming that much of a niche any time soon. For other types of work - entirely different story.
I put "100% AI-free organic content" in the footer of my blog. [1] So people see that on every page.
I'd suggest something similar.
Well here is an idea - check the actual paper contents, rather than text patterns. If you still can't tell if it's made up by a bot, it's either not a paper worth of publishing or you're not much of a peer reviewer.
https://www.nature.com/articles/d41586-021-01436-7
"Nonsensical research papers generated by a computer program are still popping up in the scientific literature many years after the problem was first seen, a study has revealed"
They say "hundreds" but the actual scale of the problem is more like tens of thousands of identified papers and probably 10 or 100x more unidentified cases.
Reminds me of that oddly prescient r/xkcd comic (https://xkcd.com/810/), pointing out the same issue for spam in online forums.
For example, if there is a specific advantage to using fluent English text, and the submitters are not fluent in English, then using ChatGPT seems like a fairly natural way to handle that. I'll bet that Google Translate is already being used for it.
Not sure that AI is quite up to actually falsifying the research (although I'll bet it will allow fakers to do much more realistic fake data).
For example, I am constantly reading prose by extremely intelligent and well-educated people, that is almost painful to read.
One of the things that ChatGPT is great for, is generating eminently readable prose. In fact, I'm recommending to my team, that we consider using it to generate some of our help dialogs.
In many cases, I wish more folks would use it.
If publishing becomes a non prestigious enough thing because anyone can GPT can do it, then at least those who do it for reasons other than "publish or peril" will be the ones remaining. Given good enough filtering exists.
The problem is, replication is expensive, thankless and most importantly rewardless busywork, so it's mostly limited to cases where a successful replication itself promises value like in the LK-99 "superconductor" craze.
I can use my friend to help draft a paper in several ways:
I can plagiarize from my friend's work, which is unethical.
Or, I can quote my friend with a citation. This is ethical.
Or, I can ask my friend to look over my paper and - without rewriting any part of it - make suggestions on how to improve it. This does not require a citation and is ethical.
Nature seems clueless about the different ways my friend (or ChatGPT) can help me draft a paper.
I do think that declaring AI assistance is a good thing, but even if an author does not do so I don’t see a huge issue here. Sure, you can have your LLM generate bogus data, but you are willing to do this you probably were unethical before LLMs anyways.
The same does not hold true for GPT.
I guess I understand the reasons for the disclosure requirement but if you're really just using LLMs to help you more quickly write some background text that you could otherwise do yourself, 1.) It's hard to see the harm beyond the tools that are in every word processor and 2.) I probably just wouldn't use the LLM at that point.