For an analogy, distinguishing AES ciphertext from random bits without the key would be counted as breaking AES (the more precise statement of this is called AEAD).
To give an example in the extreme: if you make the bias total and only make that decision based on the preceding token, there are certain token pairs that your model will never output, and this will be pretty obvious even to people just reading the text (because of any given common two-token phrase, there's a 50% chance you would just disappear in watermarked text). You can make this less extreme and more hidden by increasing the window and reducing the bias, but at the cost of reducing the signal. I don't know exactly what the tradeoff curve looks like, so it might be that you can reach set of parameters where the bias is in principle undetectable without the key but still reliably detectable for realistic lengths of text segments, but I would not assume that this is definitely the case.
The SynthID watermark scheme relies on the LLM's own logits being a map into acceptable substitutions for specific words. For example, if we have the sentence "The quick brown fox jumps over the lazy dog", the LLM might give the last word[0] as 98.5% "dog", 1% "hound", 0.5% "canine". In normal decoding we sample the logits after scaling them against a temperature parameter; in SynthID we additionally bias the logits to encode output from the watermark CSPRNG - e.g. the CSPRNG might decide to boost the third logit of this particular token to 65%.
But wait! The LLM might have selected valid synonyms for the token, but those synonyms will never be correct because this is a very well-known sentence. To humans, it's one sentence-long token, in the sense that most English-speaking humans who encounter this sentence will see the first few words and already know the ending. It thus has one bit of entropy[1]. Adding SynthID watermarks to this sentence necessarily has to change the words and thus adds more information - namely, that some process outside of the LLM itself has modified the text.
We don't need to cryptographically distinguish AES from random if both the AES (SynthID) and random (normal decoding) happens to also be constrained to form a valid English sentence.
[0] I am too lazy to account for multi-token words. Let's just assume the LLM was trained with a pretokenizer that splits exclusively on whitespace and every word in the training set got assigned a token.
[1] Compressible information
For constrained quotes like "quick brown..." the synthid watermark would be weak, but that's by design.
The AI with watermark picks its bits from a CSPRNG, then does the same transform to get the distribution. The result I think is that you can reduce distinguishing the distribution to breaking the CSPRNG.
In this example the entropy after the transform is around 0.13 bits, so the watermarking scheme cannot add any useful information. Once you have hundreds or prehaps thousands of bits of entropy, it starts to become feasible to add information into that 'spare' space.