Hacker News Reader

TGI just integrated Guidance in 1.4.3, that by itself can support both grammar/JSON/Pydantic & tool invocation/function calling.

Langchain & Llamaindex plus Fructose really need to skip the structure adherence work & move to chunking/KG generation since that's the next pain point to tackle.

View on HN

What is TGI? Is that the huggingface Text Generation Inference project on GitHub?

Yes. TGI is Huggingface's version of LLVM (some nuance, of course). LLVM also launched grammar support recently too, so we'll be looking into it.

All of these acronyms are so confusing. I'm assuming LLVM isn't the compiler tool, but searching "LLVM ai" doesn't give me any good results.

They probably meant vLLM https://docs.vllm.ai/en/latest/

ah shoot, yes I meant vLLM, sorry for the confusion, lots of comments to reply to :)

Why do you have Guidance in caps?

https://github.com/guidance-ai/guidance

or ...

https://huggingface.co/docs/text-generation-inference/concep...

or ... ?

A quick glance through these, they don't seem yet to leverage json_object on OpenAI with the word JSON in the prompt, which works wonders with the 0125 models.

Reply on news.ycombinator.com