Hacker News Reader

Comment by bartvk | Hacker News Reader

I teach and it's not a war at all, from my point of view it's actually quite simple. We use Gradework. Students upload their work in Word or PDF, and there's a percentage counter in the top right corner. If it's above 30%, which rarely happens, I pass it to the head teacher. They have a conversation with that student, which I'm told involves a talk-to for the first time, a warning the second time, and expulsion the third time (which AFAIK hasn't happened yet).

I don't see why a code review couldn't be the same.

View on HN

I looked for information on gradework, and couldn’t find anything.

From your description, it sounds like you presume the software is accurate.

I’d expect there to be peer reviewed studies about its error rate, a description of how it works, limitations, etc. For instance, if you ask for a one sentence factual response to a question, there’s no way it will be able to distinguish LLM from human. I’m sure it has other limitations.

I met someone that built one of the better packages in the “don’t copy your answers from google” space a while back.

He routinely gets pleas for help from people his software has falsely accused of plagiarism (the tool is open source and popular and maintained by others).

He generally sends strongly worded letters to educators explaining the limitations of the tool, why it’s probably wrong in this case, and that says he will happily testify against them in court as an expert witness.

I imagine the commercial vendors are less transparent, but not more accurate.

Youre letting a computer tell you what to do?

> I'm sorry, Dave. I'm afraid I can't do that.

I have no problem when a computer say's what to do, but i have a problem when a program decides what's right.

You think you can reliably identify AI text?

Anecdotally my experience with generative AI is that the valid output space is very tiny especially under a restrictive prompt, i.e. homework question, even smaller when the output also has to be a correct answer. Using a riskier decoding scheme only generates output that is paraphrastic to each other. So lexical similarity should catch most of the naive prompter.

Subjectively, if a student can knowledge-guide an LLM to generate response from a specific perspective, I think it is very debatable whether such usecase should be outright punished.

Indeed, they would just be preparing for industry in that case (guiding the LLM)

Ok, given that restrictions I can see how you can arrive at a pragmatic solution, so long as we don’t have the illusion this is actually a solution.

For example, giving it a small corpus of my previous handwritten work and asking it to respond in the same style is exceedingly easy and very, very hard to detect. Especially if you run a small algo on it afterwards to introduce your category of typos etc.

It is only a matter of time before services will pop up for students that automate/wrap this.

In day to day code review I most definitely can. I can recognise fairly well each of my team coding style, when it deviates significantly from that it's either blindly copied or comes from AI.

It’s the opinion of OpenAI that ai text detection is not possible. Do you really want to argue that you know more than they do?

Reply on news.ycombinator.com