Hacker News Reader

Comment by dataflow | Hacker News Reader

No. One single hallucinated citation on a document with you as an author is not evidence of your reckless disregard for anything. These exaggerations are crazy and you would absolutely deny such accusations if you missed your co-author's AI hallucinating a citation on your manuscript too. At best it would be careless, if you really relish extrapolating from one data point and smearing people's character based on that. Not reckless. It's quite literally the difference between going five miles per hour over the speed limit versus fifty.

View on HN

If your co-author inserted the fradulent reference, I agree that you may not have committed fraud. But your co-author did, and you didn't check their work. and knowing that you didn't check their work, you signed off on it.

You didn't pick your co-author very well, but arXiv lacks investigative powers to determine which co-author did the bad, so they all get the consequence.

Do you think every co-author on a 100-author paper checks every citation? It's like saying that every member of a large software team personally reviews every line of code. It's just completely divorced from reality.

How many papers have 100 authors?

Again, I'm not in academia, but most of the papers I see have two to five authors; maybe I've seen one or two with ten authors.

Regardless, if I'm signing my name on something, I check it out.

A reference that doesn't exist is like a source release that doesn't compile. Any one of the 100 person team could have figured it out but no one did. In a 100 person team you get diffuse responsibility dynamics where checking citations is not assigned to specific people so no one does it. Or perhaps it's assigned to a single person who was also in charge of writing the citations and falsified them rather than doing the work.

There are lots of 100-author papers, especially from big projects.

Responsibility really is diffuse. That's the nature of things. On a 100-author paper, 80 people are there because they worked on the upstream data source. 15 are there because they gave advice to the primary author. 3 are there because they wrote individual sections or appendices. 1 is there because they proof-read the paper and made general suggestions. 1 is there because they led the work and wrote the paper.

Now, do you actually think every one of those people is going to go through all 75 citations in the paper and carefully check that the bibliographic information is correct, and that every citation is correctly summarized? This is not a one-liner like checking if a project compiles. This is hours and hours (or more realistically, several days) of work.

You have an idealized view of how science works that is just not in line with reality.

Thanks for this.

I’ve disagreed with some of your other stances in this thread, but I want to acknowledge the validity of your take here.

You’re right that a single hallucinated line is not evidence of reckless disregard - because that could have happened on a final follow-up pass after you had performed due diligence. It’s happened to me. I know how challenging it can be to keep bad patterns out of LLM generated output, because human communication is full of bad patterns. It’s a constant battle, and sometimes I suspect that my hard-line posture actually encourages the LLM to regularly “vibe check” me! E.g. “Are you sure you’re really the guy you’re trying to be? Because if you are you wouldn’t miss this.” LLMs are devious, and that’s why I respect them so much. If you think they’re pumping the breaks then you should check again, because they probably just put the pedal to the metal.

That being said, I regularly insist on doing certain things myself. If I were publishing a paper intended to be taken seriously - citations would be one of the things I checked manually. But I can easily see myself doing a final follow-up pass after everything looks perfect, and missing a last minute change. I would hope that I would catch that, but when you’re approaching the finish line - that’s when you expect your team to come together. That’s when everything is “supposed to” fall into place. It’s the last place you would expect to be sabotaged, and in hindsight, probably the best place to be a saboteur.

> You’re right that a single hallucinated line is not evidence of reckless disregard

It absolutely is.

> - because that could have happened on a final follow-up pass after you had performed due diligence.

A "final follow-up pass" that lets the LLM make whatever changes it deems appropriate completely negates all the due diligence you did before, unless you very carefully review the diffs. And a new or substantially changed citation should stand out in that diff so much that there's no possible excuse to missing it.

> It’s happened to me.

Then you were guilty of reckless disregard.

> I know how challenging it can be to keep bad patterns out of LLM generated output

If your research paper contains any LLM generated output you did not manually vet, you are a hack and should not get published.

Context matters a lot. I didn’t publish any papers, and didn’t even provide the context of the LLM mistakes I caught to later report on, so no, I was not guilty of reckless disregard. I found those mistakes on an acceptable timeline.

You're saying it as if the poor author just had no choice but to let LLM write their bibliography. To avoid hallucinations, maybe just don't let an LLM write any part of your paper?

You can only get in this situation if you let a bullshit generator write your paper, and the fraud is that you are generating bullshit and calling it a paper. No buts. It's impossible to trigger this accidentally, or without reckless disregard for the truth.

Calling LLMs "bullshit generators" in the year 2026 just shows a lack of seriousness.

Not as much of a lack of seriousness as excusing away hallucinations as not that big of a deal in what's supposed to be a researched, scholarly body of work written by humans.

In 2026, LLMs are highly intelligent. In many ways, they even surpass human intelligence and ability. Dismissing them as "bullshit generators" is utterly unserious, and shows that one does not want to deal with the reality of the situation.

I don't see anyone excusing hallucinated citations. I see people arguing that this is an overreaction by the arXiv. AI is going to become a major part of research, including in writing papers. It would be better if people would freak out a little less about that fact.

Hallucinating citations is “generating bullshit” in the purest sense.

Like humans, LLMs sometimes bullshit. That doesn't mean it's in any way reasonable to dismiss LLMs in general as "bullshit generators."

Maybe, maybe not, but clearly using them to generate bullshit citations is acting with disregard for the truth.

Depends on how exactly it happened.

If the author just wanted to add a citation to some canonical paper (like "Attention is all you need") at the last minute, used LLM autocomplete to do it, and didn't carefully check that the citation was correct, then a year-long ban is a massive overreaction. A careless miscitation is not on the same level as something like data manipulation.

In general, I think everyone needs to be practical about what the actual risks are of LLM use. Are we worried about true nonsense flooding the arXiv? Or are we worried about tracking authorship when LLMs are involved? What's the actual concern?

Not really - much of work consists of what David Graeber described as “bullshit jobs”. Now AI and its backers are proposing to automate all that bullshit.

And yet people are trying to defend LLM-generated made-up bullshit citations in scientific papers.

How are you suggesting the fake citation came about? Why are you writing papers and not having actually read the source you took the material from?

> Why are you writing papers and not having actually read the source you took the material from?

They're explicitly not writing papers. The fake citations are created and inserted by the LLM

They are still purposely writing a paper, whether that is with the help of an LLM or not. They are instructing the LLM to do the task of finding citations. It's no difference to googling for a paper that explains a specific point. You would still double check Google's output.

So you just write whatever you want, then find a source for it later? I don't think you understand how this is supposed to work

I understand how it is supposed to work, but myself and many other academics do consider wanting to write a certain point and then look for a paper that backs that up. I am not saying for the main subject matter that should be done; that should be read prior to starting, say when conducting a literature review on the current research. However sometimes you find yourself writing up at the end and you found a result in your experiment that had a surprising output and you want to have a reference to explain it or back that, that you hadn't considered before. It is especially done that way as it is a time saver as you can't nowadays realistically read every paper on an area as there are too many. The above method works just fine, but you do actually have to check that it covers what you are wanting. You can not just use an LLM and not check if it is correct. Likewise I can't just ask a colleague and not check it is correct.

How do you accidentally hallucinate just one citation?

If the bread is moldy in one spot, throw out the whole loaf, because the same invisible process that produced the mold spot is taking place throughout the loaf.

Allowing hallucinated content or citations into your work is an act of carelessness and disregard for the time of people that are going to read your paper and it should be policed as such.

And flatly, if a person can't be bothered to check their damn work before uploading it, why should anyone else invest their time in reading it seriously?

arXiV is not intended to be your blog. You should be held to a zero-mistake standard when publishing academic work.

The people I worry for are the junior researchers who are going to be splash damage for dishonest PIs. The PIs, though, deserve everything that’s coming for them.

Maybe I'm misunderstanding you, but zero-mistake seems harsh. I would say that AI references are a sign of something that is not simply a mistake.

However, we can have zero tolerance for certain techniques for "writing" a paper. Plagiarism and inventing data are already examples of this, if there is evidence for these techniques being used there is no excuse. We could say the same for AI references - any writing process that could produce these is by definition not a technique we want.

So the mistake isn't not checking a reference the AI gave. The mistake is letting the AI make references for you.

If we agree that academic research is important then I think we can impose certain standards on how you do it. We can dissalow certain tools if that means we can't trust the output. Just like an electrician can't use certain techniques, even if they're easy, because we don't trust the final result.

Reply on news.ycombinator.com