StructuredRAG: JSON Response Formatting with Large Language Models

Interesting paper, but their reason for dismissing constrained decoding methods seems to be that they want to academically study the in-context setting.

For practitioners, using a framework like Guidance which forces the models to write valid JSON as they generate text solves this trivially (https://github.com/guidance-ai/guidance)

For json in particular these frameworks have functions that take in json schemas or pydantic schemas https://guidance.readthedocs.io/en/latest/generated/guidance...

> It may additionally be promising to test Chain-of-Thought (CoT) prompting strategies. This entails adding a “rationale” key to the model’s output such that the additional reasoning improves the performance of the model. However, this will require the output to be a composite object with the additional "rationale" key and string-valued response. Our results suggest that this additional output structure may result in lower success rates. Similar in spirit to structured decoding methods, it may also be helpful to prefix the end of the prompt with “{“ or use the key, such as “{“paraphrased_questions”: [".

For most of my prompts I want CoT.