Hacker News Reader

Comment by J_Shelby_J | Hacker News Reader

So to explain this another way:

After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token?

Very slick!

View on HN

You would also need to keep generating until the whole string is valid. And what if it gets caught in a loop?

Not sure how this can really guarantee 100%

> And what if it gets caught in a loop? Not sure how this can really guarantee 100%

It's not great but after some timeout you can just set the mask to only include closing brackets.

You would still have to ensure balancing somehow. Both "]" and "}" are valid "closing brackets" and the correct one to choose is context-dependent.

You can determine which brackets you need in which order by parsing the incomplete json which was generated so far.

That won't do it, also need to close other stuf

{"this": "is valid json so farrrrrrrrrrrrrr

But yeah the general idea makes sense. Once you hit a timeout, change the mask to things that will close existing open things in a valid manner (}, ), ], ")

Same problem with normal sampling - if it doesn't pick the <end> token, you're stuck generating until you hit some stopping heuristic (max tokens, timeout, etc.)

Indeed. And we're able to update the mask with a dictionary lookup instead of looping over the entire vocabulary (slow!).

You also need some kind of beam search or rejection sampling since JSON tokens to not exactly correspond to logits.

edit: They describe this more carefully in the paper.

It’s actually a very old trick. Lots of libraries do this. idk what’s the big deal about this one.

Perhaps I didn’t explain clearly enough in the original post?

Reply on news.ycombinator.com