ReLLM: Exact Structure for Large Language Model Completions | Hacker News Reader

ReLLM: Exact Structure for Large Language Model Completions

matt-rickard.com

26 pointsrckrd13 comments

This is clever but seems to work through brute force and my guess is that results should be accompanied by some kind of confidence measure. You're forcing the model to choose a path it wasn't particularly interested in going down.

In the ideal case, such as when you ask for a json array, the first "[" token you enforce will have a relatively high probability and forcing it to go down that path will give you good results.

In the dangerous case the model doesn't have a good structure-compliant completion to your prompt and the regex you supply forces the model into a path of extremely low probability and you get trash results.

That sounds like a real risk but also the kind of thing you would need to implement a solution for anyways.

It seems like there's two clear paths:

- Allow the model to complete whatever it wants and then anneal the structure into compliance - Force the model into a compliant structure and then anneal the quality

I think both options can make sense in different cases.

One case I'm thinking about where the second option feels simpler is when you want to implement a boolean function using a language model. I'm imagining a probability distribution that looks like:

- (40%) the answer is true - (39%) True - (21%) False

In this case it seems significantly more straightforward to force the model into completing T or F. I guess you then run into the "dangerous case" where you have

- (40%) the answer is false - (39%) True - (21%) False

(author here) That's interesting! Maybe there's a way to quantify the cumulative probability of the squashed tokens (i.e., if you constrain to 'true' and 'false', what's the distribution of the other tokens).

For now, this is a good way to make sure that I can parse the output reliably in the minimal amount of completions (instead of looping until conformant).

Is similar to this work on constrained text generation:

https://github.com/hellisotherpeople/constrained-text-genera...

Also https://lmql.ai/

I think LMQL is the best example I've seen of the "forcing the LLM to walk a certain path" technique. It's a DSL written in Python for Python though, so it kinda constrains it's utility.

LMQL dev here :) Can you specify what you mean "constrains its utility". Are you looking to use LMQL from different languages? We are currently thinking about expanding to other ecosystems, so it would be interesting hear your thoughts.

This seems extremely similar to https://github.com/newhouseb/clownfish

Similar strategies with the logitsprocesor. It is a more generalized version that's not just constrained to JSON parsing, but any regex. JSONformer/clownfish try to parse the types syntactically.

A regex is a better fit for a different class of problems. You might implement a JSONformer/clownfish with this instead.

Makes sense. I wonder if it would actually make more sense to just plug a function that generates the mask because in general it's just:

  fn complete(input, masker):
    completion = input
    while (not done yet):
      mask = masker(completion)
      completion = complete(complete, mask)
    end
    return completion
  end

is that accurate?

There's probably a better API that wraps generate, but there's a bit more work than the logit mask.

You have to go one token at a time, otherwise the masking becomes combinatoric rather than linear (two tokens at a time -- need to generate all two token pairs, etc.).

But otherwise, that's what the code does! https://github.com/r2d4/rellm/blob/main/rellm/rellm.py#L21

How’s this compared to ReLM: https://arxiv.org/abs/2211.15458

My first thought was can you get the LLM to generate the pattern first, and then the completion?

That would almost certainly work to improve reliability but you would probably need something like this anyway, wouldn't you?

In general, it seems like building a production-ready LLM-based application requires _a lot_ of these little tricks and methods to hammer results into shape before sending them downstream.

Reply on news.ycombinator.com