Hacker News Reader

Comment by Art9681 | Hacker News Reader

If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone else, it must cross over the 50% threshold. This is the problem.

Cognitive surrender.

View on HN

> the probability of the detection tools flagging completely human generated text as AI generated is non-trivial

How does that follow? AI-generated text is already not a perfect emulation of human writing. There's lots of room to affect it laterally without changing the level of quality.

As I understand it, LLMs with temperature >0 can select from many possible outputs. All they're doing is limiting the possible outputs to ones that contain this pattern. I don't see any reason why the quality of that subset should be lower than average. The very best outputs will likely be eliminated, but so will the very worst.

To sample from the probability distribution you already need random numbers.

If you get your random numbers from a cryptographic PRNG, then to notice the difference between that and 'real' random numbers even in theory, means you need to break the cryptography. In practice, your gut feeling about how good some text is won't break modern cryptography.

That's an interesting problem to discuss, but unfortunately TFA spends no time discussing that.

Yeah. And the detector can’t even tell you “X% chance this is watermarked” because it doesn’t know the input distribution. It can only tell you “Y% chance that an unwatermakred text would score this high” and how many history professors understand Bayes rule well enough to understand the distinction?

Worse, what will academic institutions decide is the threshold for detecting AI generated work. If you have a false positive how do you prove it was a false positive or we all just trust the watermark detector over the student saying "I swear I did it all by my self"

I don't think that's true. I think it's a binary 0% or near 100% probability of a watermark having been detected; the more changes to the text having been made after the text was output by the LLM and the less leeway the LLM had for probable word choices, the longer the passage necessary to see it.

The "problem" is that seeing the watermark doesn't mean that the person claiming to be the author didn't make extensive changes to the output of the LLM, or that the LLM wasn't simply the final editor of something that the author had put a lot of work into.

> Cognitive surrender.

I don't know what this means. It's just drama. Don't let the LLM write for you and this is not a worry. I'm not worried about the poetry of LLM output being subtly adulterated.

No, watermark detection is not binary, you get a real number. You decide on a threshold when looking for the watermark. This is the problem - by random chance, some human text will be detected as watermarked. You can turn the detection threshold up until it guarantees <0.001 false positive rate at the expense of higher false negatives, but seems inevitable that someone gets wrongly flagged.

Reply on news.ycombinator.com