Hacker News Reader

Comment by pkulak | Hacker News Reader

It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking, it's a better LLM.

View on HN

It's a very unintuitive algorithm, and is pretty clever.

I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4

But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.

What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?

It would be picked 10% of the time with watermarking.

The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.

It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.

What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time?

I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.

To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability):

> SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).

Unless you’re at 0 temperature, there is no single token it would have chosen. It’s always picking one of multiple randomly according to a probability distribution.

Give me an example how would you watermark a single short sentence like "I like turtles"?

Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text

It's like the sudden change of a language style and its verbosity didn't happen recently.

To random words you pick and provide a sufficient amount of text to vary with random number without losing its meaning you need a text with high entropy.

Nothing about watermarking would require padding the response length with pseudo-intelligible Claudese. Regular filler would work fine.

Also, it would probably provide higher entropy to write normal human-sounding English instead of reusing a repetitive grab bag of load-bearing phrases. This theory doesn't really make any sense.

No, it basically uses a fixed seed for the random number generator to generate tokens. The generated random sequence is just as random as it would be with a random seed.

Unless you’re running at temperature 0, there’s not one single token that the model definitely would have chosen each time.

Reply on news.ycombinator.com