Is there a reason we won’t have LLMs that only speak in JSON? These JSON hacks are clever and cool but feel like they’ll be obsolete in 6 weeks.
Technically speaking it's pretty to force the model into an valid JSON-schema if you have access to the inference autoregressive loop and the logit activations. You can either force known areas of the template to a prefixed template, or fill in the basic JSON wrapper and force the model to choose arbitrary keys and values.
Imagine it might come down to how much of their usage is on generating text vs generating structured prediction payloads.
Why do you think they will be obsolete in 6 weeks?