I highly doubt it's the model that does this... It's very likely code injected into the token picker. You could put this into any model all the way down to gpt-2.
My assumption is if that's all this is they would have done it a long time ago though.
There are quite a few open source implementations of this e.g. https://github.com/outlines-dev/outlines
1. There is always a valid next token.
2. This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm.
The latter isn't too obvious, and may in fact be (very) false. Look up maze generation algorithms if you want some feeling for the effects this could have.
If you just want a quick argument, consider what happens if picking the most likely token would increase the chance of an invalid token further down the line to nearly 100%. By the time your token-picking algorithm has any effect it would be too late to fix it.
If you want to be really clever about your picker, a deterministic result would blat out the all the known possible strings.
For example, if you had an object with defined a defined set of properties, you could just go ahead and not bother generating tokens for all the properties and just tokenize, E.G. `{"foo":"` (6-ish tokens) without even passing through the LLM. As soon as an unescaped `"` arrives, you know the continuation must be `,"bar":"`, for example
> This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm.
It absolutely will. But so will adding an extra newline in your prompt, for example. That sort of thing is part and parcel of how llms work
That said if you know all valid prefixes in your language in advance then you can always realise when a token leaves no valid continuations.
> It absolutely will. But so will adding an extra newline
A newline is less likely to dramatically drop the quality, a greedy method could easily end driving itself into a dead end (if not grammatically then semantically).
Say you want it to give a weather prediction consisting of a description followed by a tag 'sunny' or 'cloudy' and your model is on its way to generate
{
desc: "Strong winds followed by heavy rainfall.",
tag: "stormy"
}
edit: oh actually, we do sort of know -- they call out jsonformer as an inspiration in the acknowledgements
This might be even more pronounced when the output is restricted more using the JSON schema.
So the heavy lifting here was most likely to align the model to avoid/minimize such outcomes, not in tweaking the token sampler.
FWIW (not too much!) I have used llama.cpp grammars to restrict to specific formats (not particular json, but an expected format), fine-tuned phi2 models, and I didn't hit any issues like this.
I am not intuitively seeing why restricting sampling to tokens matching a schema would cause the LLM to converge on valid tokens that make no sense...
Are there examples of this happening w/ people using e.g. jsonformer?