Comment by crazygringo | Hacker News Reader

Comment by crazygringo | Hacker News Reader

It seems totally unclear whether ChatGPT is being used to invent fictional methodology or literature review etc., which is extremely unethical, or whether it's simply being used to check and fix grammar and tone for non-native speakers, which is perfectly fine.

But regardless, this is extremely troubling in terms of quality. The fact that "regenerate response" gets left in tells us that even if it's just for grammar correction:

1) The authors aren't even minimally reviewing if the ChatGPT version didn't accidentally change the meaning of the sentences

2) Peer reviewers clearly aren't even reading the paper

3) The journal isn't even reading the paper to introduce a minimal level of quality control

ChatGPT seems to be the smallest problem here. And thankfully, it is revealing much more serious problems about quality checks in academia at all.

The question is, will anything be done about it? Not about ChatGPT, but about the horrifically irresponsible processes that would let these slip through?

EDIT: Oh, wow. Lack of coffee this morning. I read that as "fictional mythology", NOT "fictional methodology."

Ignore me. :)

-----

I want to pull apart one specific thing you mentioned, essentially:

> It seems totally unclear whether ChatGPT is being used to invent fictional methodology [...], which is extremely unethical

In the context of generating something fictional, why is using ChatGPT unethical?

One reason I'm asking: I have always been a big fan of NaNoWriMo[1] (National Novel Writing Month, every November). This year, I'm interested in exploring how to use ChatGPT -- not necessarily to generate a ton of prose, but to essentially help flesh out some plot ideas that I've had.

Part of the back and forth I've had with it is coming up with some interesting scenarios for my story that I hadn't really thought about and I'm keen on exploring a bit more.

[1] https://en.wikipedia.org/wiki/National_Novel_Writing_Month

Because this fiction is presented as part of a scientific paper....

I am a ding dong. I read that as "fictional mythology" and my mind immediately went down a different rabbit hole. Yes, I see now why that would be unethical.

its unethical because its not being published as fiction, but of science.

Methodology isn’t supposed to ever be fictional is the problem.

But ChatGPT isn’t deciding to use fictional methodology in its paper, it’s being asked to generate a fictional methodology and complying.

The human is the problem in that scenario, not the LLM. The unethical choice happens at the point of deciding to lie about methodology, before LLM is involved.

It may seem like a meaningless distinction, but it’s the difference between LLMs causing trouble and people causing trouble, which is important for the larger debate.

Not meaningless, but a little obvious. I don't think anyone was accusing ChatGPT of being duplicitous.

I think one problem with LLMs and the hype around them is that people are placing far too much confidence in them, and therefore misuse them.

That it's even conceivable that people might blame a tool for unethical behaviour is an indicator of this.

The idea that an LLM would be able to correctly generate your methodology for you is another.

I find ChatGPT simultaneously incredible and terrible. It has no concept of what it knows and what it doesn't, so there's nothing I would trust it to do without a human checking the output.

This reminds me of when I asked it to generate a fake research paper describing how my dog is the best in the world as a joke and it refused because apparently creating fake research papers is unethical, even when it’s an obvious joke.

I asked again in a new chat and it made me a really nice one with funny things like a formula showing the amount of force he uses to open doors and fake data proving statistically that he is the best

Yes, I agree. I misread the word. My apologies!

They mean generating fictional methodology and then passing it off as real in a scientific paper is unethical. This is also unethical if you don’t use ChatGPT, of course

Ah! Misunderstanding on my part. I see. Thank you!

If I recall correctly, a number of years ago, plenty prior to ChatGPT, there was an incident with a scientific journal (don't recall the name though). Perhaps others here may know of this incident too...

So apparently someone submitted a paper that was entirely nonsense. This was intentional but not explicit as it was just a stunt. The paper had all the right fancy/technical words, the right structure, likely even images, and apparently a convincing conclusion. But it was all about nothing, nothing substantial at all, nada. Had a peer reviewer actually read the paper, it would have not been published and summarily tossed. But it did go print and oh boy, the journal's board/powers-that-be were pissed. I didn't follow the story much past that point but I think something was supposedly proposed to prevent this.

It was a little bit more complicated than that. https://en.wikipedia.org/wiki/Sokal_affair

I guess that would be it! Thanks for pointing out that Wikipedia entry. I was in college at the time majoring in Physics, and yep, that was one of the headline news among a journal or two that arrived there. Good memories, indeed...

What was more complicated? I read the wiki article and it basically matches the description one to one?

I can confirm using chatGPT for paraphrasing abstract, conclusion etc but I always edit it to make sure that it conveys the same meaning and has no errors.

Reply on news.ycombinator.com