Guidance: A guidance language for controlling large language models

This is how we implemented it anyhow, with some more parameters to control how that all works (and the LLM params) at each "pause" point. The _neat_ part for us was that a template helper could make use of the partially generated content. Hadn't thought about that before for a templating engine, but was trivial to implement in the end

Each expression becomes a new inference request, so it's not a single inference pass. Because each subsequent pass includes the previously inferenced text, the LLM ends up doing a lot of prefill and less decode. You only decode as much as you actually inference, the repeated passes only end up costing more in prefill (which tend to be much faster tok/s).

To work with chat tuned instruction models, you can basically still treat it as a completion model. I provide the previously completed inference text as a partially completed assistant response, e.g. with llama 2 it goes after [/INST]. You can add a bit of instruction for each inference expression which gets added to the [INST]. This approach lets you start off the inference with `{ "someField": "` for example to guarantee (at least the start of) a json response and allow you to add a little bit of instruction or context just for that field.

I didn't even try with openai api's since afaict you can't provide a partial assistant response for it to continue from. Even if you were to request a single token at a time and use logit_bias for biased sampling, I don't see how you can get it to continue a partially completed inference.

[1] https://github.com/gsuuon/ad-llama

I would not expect it to make a difference in your current applications. Getting JSON is all about the model, training, and prompt, in that order

If you are looking for low-hanging fruit to improve your JSON responses from LLMs, fine-tuning will likely get you the most bang for your buck. Start from a coding model like codellama, code-bison, or starcoder

It's obviously extremely valuable if you're doing anything with the LLM output other than displaying it as a block of text to the user, or if you care about the output format at all.

LangChain: I found having a framework useful to ramp up people without prior LLM exposure, in an open-ended experimental space. The library covers many usecases and gets people thinking. But honestly their documentation is somewhat lacking for that purpose (stale text, shallow examples). Personally, coming from a search background I was able to DIY semantic RAG in the time it took to figure out how to do the same thing in LangChain.

You can get the same thing with Go text/templates by adding chat function(s) as custom a helper: https://github.com/hofstadter-io/hof/blob/_dev/lib/templates...

As a developer of these things, I don't get why they want to put so much effort into the mundane parts rather than focusing on the interesting parts. These things are mostly just the same as any other workflow or API call: https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/cmd... (unless you get into the python and (i.e.) start messing with the logits or token probabilities)

https://github.com/simonw/llm

How often does a project need to release to not be considered dead? It's only been 10 weeks, in the summer, at the peak of vacation time

Look at the most recent commits, they are setting up new governance, which likely took more than 1o weeks to work through the bureaucracy of Mircosoft

I thought guidance was smart, but LMQL seems brilliant as it merges pythonic constructions with LLMs (I think it may be an outright superset of python with LLM functionalities?)

It's predicated off a paper as well : https://arxiv.org/pdf/2212.06094

Somehow, the VCs and investors made us think it was cool to be working for them rather than our users

https://github.com/microsoft/guidance redirects to https://github.com/guidance-ai/guidance now.