Comment by hex4def6 | Hacker News Reader

Comment by hex4def6 | Hacker News Reader

Parent Full thread

Go Aggies! Good to see my alma-mater getting known for something other than pepper-spraying sitting protesters in the face.

Turnitin advertises the feature as "98% accurate". That's a pretty terrible accuracy rate for potentially destroying someone's life.

The utter hypocrisy of having to write "ethics / impact" blurbs for class projects, when the school can seemingly implement this sort of life-changing technology without a long trial period and carefully designed guard rails around how to interpret the results is galling.

The best case scenario is that there are so many false positives that they get recognized for the garbage they are, but I think in reality there may be a small population of students who are unlucky enough to write in a way that gets flagged significantly more often.

How do you defend yourself against that sort of situation? "Well, your classmates didn't get flagged. Why did you?"

The problem with this technology is that there is no concrete evidence that you can use to prove / disprove the accusation. There is no single piece of plagiarized content that you can dig up and point to as the smoking gun.

The ethics of letting this technology go live is breathtakingly stupid. I'm concerned this sort of laziness is a harbinger of things to come, and it fills me with dread.

>Turnitin advertises the feature as "98% accurate". That's a pretty terrible accuracy rate for potentially destroying someone's life.

Assuming the 98% applies to both false negatives and positives:

For a given 20k person school, assuming they each write only one paper a year that's run through this, an administrator gets to expel 400 students a year!

What fun!

The problem is, that's actually sort of the best-case scenario. Obviously, the protests and insanity of that situation would cause it to be quickly scrapped.

The more insidious issues is you have a sub-population of students that write in the "wrong" way. That means the average student might never get a strike, but this unlucky sub-population might get multiple strikes. That would seem to point to the results being "correct" -- after all, you'd expect a cheater to cheat multiple times.

If this population skew is significant enough -- say, 1% of the population writes in the "wrong" way, and the multiplier effect is 10x as strong for them -- you might never get a flag for a normal student. But what you're measuring is not cheating, but deviation from the norm in writing style.

I'll just point out that college students papers are exactly the sort of thing these AI chat things also do. Take a body of preexisting material and distill it into an essay. Really seems to me this is moving the bar as to what counts as cheating rather dramatically.

Memory that pops up is Barton Fink being told to write a script for a wrestling film.

Also sort of topical in high school had a quiz on a short story that I had not read. But I had read some of the authors other short stories. So I started out with 'While I failed to read 'story title'. Based on the author and the title I think the story is about bla bla. And the teacher took off 2 pts for not reading the story first.

98% accuracy?

Haven't these people all taken statistics course? Or at least enough of them. Just to read that one number and do some rough estimations of what it means in any meaning...

You can detect it. You interview the student about their paper. If they don't know or understand what they wrote, then either they cheated or they may as well have cheated because they didn't learn anything. If you have enough money (they do), make the interview part of the assessment for all students.

Reply on news.ycombinator.com