LMQL is a programming language for language model interaction
lmql.ai
lmql.ai
LMQL seems to allow for fine grained control of prompts as the results are generated. The constraint system is super interesting. I'm going to have to read the paper, I'm interested in how they're achieving token costs savings with a system that looks like it should increase it.
LMQL`s efficiency gains can be attributed to close supervision of the generation process, as the token masking via constraints is directly integrated into the decoding loop and happens on the token level. Compared to text-based high-level APIs, this means you can save a bunch of useless continuations the LM will produce, that further down the pipeline you have to discard, as you may want to enforce constraints, insert some follow-up instruction, or tool execution result.
For stuff like llama.cpp where you can control the inference loop directly yes, this sort of thing can make sense albeit maybe more as an API rather than a programming language, but for OpenAI where you can't interact with the loop as it runs it feels like it'd get expensive really fast. I guess for research that doesn't matter?
Cost is definitely a dimension we are considering (research has limited funding after all :) ), especially with the OpenAI API. Lock-step token-level control is difficult to implement with the very limited OpenAI API. As a solution to this, we implement speculative execution, allowing us to lazily validate constraints against the generated output, while still failing early if necessary. This means, we don't re-query the API for each token (very expensive), but rather can do it in segments of continuous token streams, and backtrack where necessary,
This is still more expensive than doing all in one request, but it is an inherent limitation of the OpenAI API, and not LMQL. On the upside, you gain more control, scripting and constraints, even with OpenAI models.
Ideally, some program representation of a scripted prompt like LMQL queries could be send over to the inference service, and be executed locally with full model access. This way, model vendors would not have to expose their models fully (e.g. to protect against distillation), but API users would gain a lot more control and efficiency. Alternatively, of course, better open source models with full access to logits are the ultimate solution, which is also the context in which LMQL was initially conceived in.