Show HN: Grammar Generator App for Llama.cpp | Hacker News Reader

Show HN: Grammar Generator App for Llama.cpp

grammar.intrinsiclabs.ai

19 pointsaduffy6 comments

llama.cpp added context-free grammar guided generation functionality. It requires passing a file in a derivative of BNF notation, which gets messy very quickly for things like JSON.

To improve the experience, we built a small compiler from TypeScript interfaces to the grammar file format and have it hosted in a little browser app.

See more in discussion at https://github.com/ggerganov/llama.cpp/discussions/2494

Open articleView on HN

Can you add examples on how to use this because I have honestly no idea how to use it

Thanks for the feedback. There's an example in the linked GH discussion but let me provide a quick ETE example here, centered around parsing structured information from a hypothetical shipping company email.

1. In the app, write a TypeScript schema for this, one good example could be

  interface DeliveryInformation {
      /* Tracking number for the delivery */
      tracking_number: string;
      /* Status of the delivery, one of "preparing", "out-for-delivery", or "delivered" */
      status: string;
      /* Weight of the package, e.g. "2oz" or "3lb" */
      weight: string;
      /* submission date time representation */
      submitted_ts: string;
  }

If you click Generate you'll see it generate a context-free grammar looking text, which is what llama.cpp reads. Click the download file to save it as grammar.gbnf

2. Grab a model for llama.cpp, e.g. this quantized Llama2 chat model: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma...

3. Grab the prompt in this gist (https://gist.github.com/a10y/d926039eee63cc2bcaf6345f9a419e3...) and save as prompt.txt

4. Compile a recent commit of llama.cpp and run the following command

  ./main -m ./models/llama-2-13b-chat/llama-2-13b-chat.ggmlv3.q8_0.bin -f prompt.txt -c 4096 -n 1000 -t 1 --temp 0 --grammar-file ./grammar.gbnf

EDIT: fixed some typos

You might want to add an appropriate modifier before “grammar” on your top page to indicate your domain.

Being a natural-language person, I thought your tool might be something that, say, generates rules for verb conjugations from a corpus of a human language.

Small Nit: "This grammer can be used during LLMs generation."

I didn't know Llama.cpp had this grammar feature, so this is interesting.

It's a relatively new feature, see

https://github.com/ggerganov/llama.cpp/pull/1773

Fixed! Thanks.

Reply on news.ycombinator.com