Yes, language is like that, at least the kind of language produced by LLMs. All LLMs produce a probability distribution at each token. If you run the LLM multiple times with the same prompt you will observe it generate different responses. Using the watermarked prng does not change the distribution.

When generating tokens that might be critical to the tone or grammar or correctness, the probability distribution might be 99% on a certain token. In these cases, with or without watermarking, the output will almost always be that same token. E.g., if you ask "please output the exact word watermelon", the LLM will output watermelon with 99%+ probability even with watermark (i.e., the output won't actually be detectable as watermarked).

   Language is different; the tone changes when you change any word or even just punctuation.
   Language doesn't work that way -- moving the placement of even a comma will affect its tone.
   However, language isn't like that: even if you only drop a single piece of punctuation, that can impact the overall meaning of a sentence.

There are many ways of phrasing things that are, for all practical purposes, functionally equivalent.

Putting aside the way you're saying it, your comment has a valid and common misunderstanding of LLMs.

LLMs don't just naturally output a single suggested word (or token) each iteration. Instead, they output a value (roughly, a probability) for every possible word. It seems obvious to simply pick the top (i.e. best) suggestion each time. Then your objection makes sense: watermarking would violate this.

Of course people have tried this! The problem is, in practice this makes the LLM much less "creative" than if you randomly pick one of its suggestions (weighted by the numbers it assigned them). You can artificially increase the value higher-value outputs to reduce the chances of it saying something really odd, and this parameter is called "temperature". A higher temperature allows lower-probability choices (therefore seemingly more creative but perhaps less accurate) and a lower number vice-versa. Either extreme works poorly, and picking a good number is part of optimising an LLM.