Hacker News Reader

Comment by IAmNotACellist | Hacker News Reader

I'm struggling to understand what he's talking about. Starting with "passing up," did he invent this terminology? The only input you have to an LLM is the prompt, which gets tokenized. And if you were to send DCFG rules or a compiled version of it as part of the request, how would that fundamentally alter the way that the tokens are predicted? If the model predicts something that doesn't conform to the grammar you require, is he proposing re-prompting until it gets it right?

View on HN

You have more inputs to an LLM than just the prompt. For example, people commonly pass parameters which control the sampling of tokens.

Implementing grammar based sampling does NOT require "re-prompting until it gets it right". Imagine a point in time when the LLM is generating some particular token. Which token will it produce? To decide that, it evaluates and assigns a score to each potential token. Then it chooses one of these options based on some rules. Rules could be as simple as "pick the token with the highest score". That is called a greedy strategy. Usually more complex strategies are used and they typically have some randomness. That is called sampling. You can imagine a grammar based sampling strategy to force specific tokens at specific positions in the output, for example, to close a bracket in json.

I think he's proposing this kind of API:

    POST /openai/gpt4
    {
        "prompt": "The address of the White House",
        "sampler_wasm": "base64 encoded WASM binary blob here"
    }

That WASM would be a program that you write yourself that is run as part of the tokenizer - so it could be a grammar but it could be anything else too.

It's WASM which means it can be safely and performantly run in a sandbox by the OpenAI servers as part of their execution of your prompt.

I think you mean "run as part of the sampler," the tokenizer (and tokenization) is fixed for a given model. The sampler blob would basically:

1. Modify the output token probabilities to fit any arbitrary use case

2. Perhaps do trigger some sort of backtracking / beam-search

(I'm not Grant but we've chatted on twitter and built similar things)

Yes, I meant sampler, not tokenizer.

The model returns probabilities across the full set of tokens. This restricts the tokens to those that conform to the grammar, and samples from those

This reminds me of something i did once using a 1d CRF at the output of a sequence model. I set "impossible" transitions to -inf so that transitions between certain state pairs were simply never predicted. I could imagine executing something like this that is conditional on the current grammatical context.

On fact it's a bit surprising to me how little I see CRFs mentioned in the context of language models. They are useful whenever you want to model or learn transition probabilities.

Reply on news.ycombinator.com