Hacker News Reader

Looking at the playground, it appears the few shot examples in the prompt and CFG are duplicative. What is the relationship between the two?

When you say in another comment that using OpenAI functions to output JSON is a waste of tokens, how are you generating the JSON output? And why do your prompts then include few shot examples of JSON objects?

View on HN

The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time.

We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.

When you say “our model”, are you using a custom LLM for completions vs OpenAI or other LLM vendor?

Custom LLM—hence the self-hostability.

Reply on news.ycombinator.com