remilouf

Rémi here, really cool! It kind of turned into a rabbit hole on my end: https://github.com/rlouf/sigil

Of course: https://github.com/rlouf/sigil

It was indeed inspired from my IRC days :)

Original author here, the project has evolved quite a bit since then, you can follow here if that interests you: https://github.com/rlouf/sigil

(The $7k was sarcasm)

Author here. Sorry my writing is tedious. Next time I’ll use AI to make it more readable.

> Ironically LLMs solve the MxN problem he's complaining about

Enlighten me please

Ooops sorry

Author here. You're right, it's not a hard problem, but a particularly annoying one.

I haven't always done this, and the knowledge base used to visibly degrade over time. Reviewing a PR does not take a long time, maybe a few minutes, and this compounds over time.

This is actually pretty funny.

That’d be a pretty inefficient way to generate bullshit at scale

LLM evaluations are very sensitive to the details of the prompt's structure. This post shows how using structured generation reduces the results' variance and the ranking shifts.

Looks like it’s quite the opposite: http://blog.dottxt.co/performance-gsm8k.html

What do you mean by "semantic dimension"?

That whole structured generation line of work looks promising. I hope someone else takes this and runs evaluations on other benchmarks. Curious to see if the results translate!

Awesome work! I am really impressed by how much structured generation improves model performance.

This article presents a way to make structured generation with LLMs much faster than standard generation, but what I find most interesting is how it highlights the issues that tokenization entails towards the end.

We already support regex-guided generation in the library, and could easily make an API to serve this as well if that's a feature people want!

It is currently limited by the time it takes to build the index. There are obvious optimizations we can apply to this, however in a production setting it does not matter much since you only need to build the index once for each (schema, vocabulary) pair.

You mean nested JSON? It's totally possible.

You can definitely let the model improvise by defining `weapon` as `Union[Weapon, str]` if that's what you're asking.

It's limited to 300 logit biases at a time. Knowing GPT4's vocabulary is ~100k tokens it's not nearly enough to get reliable guided generation. Although it could work in some cases, and another advantage of this work is that we can determine that before generating.

Sorry for misrepresenting your work. Thank you for correcting me and the explanation. Will take a closer look.

Regex-Gen is implemented in all generality in the library (minus some constructs that we still have to add). JSON is merely an application.

You can read https://blog.normalcomputing.ai/posts/2023-07-27-regex-guide... for a more detailed explanation of how it works. Should answer your question :)

Thank you for the pointer. The best part of posting on HN is the long list of related work you get in response.

The Finite-State Machine we walk on during the generation process does not suffer from this problem so we can still output correct JSON, if that’s what you’re asking.

I’m sorry that our software made you so angry. It was a side project led by two people independently from the rest of the company.

They’re checking regex partial matches for each possible completion, which is intensive indeed. You can look at the Figure 2 in our paper (link in original post) for a simple comparison with MS guidance which shows the difference.

IanCal said it all. But for alternative approaches that also use LLM (with miniKanren) you can check https://arxiv.org/abs/1809.02840

We can add an integration to llama.cpp, please open an issue on the repo if you’re interested!