Hacker News Reader

Comment by frabcus | Hacker News Reader

It's a very unintuitive algorithm, and is pretty clever.

I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4

But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.

View on HN

To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability):

> SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).

What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?

It would be picked 10% of the time with watermarking.

The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.

It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.

What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time?

I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.

Reply on news.ycombinator.com