We just use openai function calls (tools) and then use Pydantic to verify the JSON. When validation fails we try the prompt again.
We instead had to write a parser to catch small mistakes like missing commas, quotes etc, and parse content even if there's things like reasoning in the response, like here: https://www.promptfiddle.com/Chain-of-Thought-KcSBh
- retry the request, which may take 30+ secs (if your LLM outputs are really long and you're using something like gpt4)
- fix the parsing issue
In our library we do the latter. The conversion from BAML types to Pydantic ones is a compile-time step unrelated to the problem above. That doesn't happen at runtime.
Python: https://github.com/BoundaryML/baml/blob/413fdf12a0c8c1ebb75c...
Typescript: https://github.com/BoundaryML/baml/blob/413fdf12a0c8c1ebb75c...
Snippet:
async def test_dynamic():
tb = TypeBuilder()
tb.Person.add_property("last_name", tb.string().list())
tb.Person.add_property("height", tb.float().optional()).description(
"Height in meters"
)
tb.Hobby.add_value("chess")
for name, val in tb.Hobby.list_values():
val.alias(name.lower())
tb.Person.add_property("hobbies", tb.Hobby.type().list()).description(
"Some suggested hobbies they might be good at"
)
# no_tb_res = await b.ExtractPeople("My name is Harrison. My hair is black and I'm 6 feet tall.")
tb_res = await b.ExtractPeople(
"My name is Harrison. My hair is black and I'm 6 feet tall. I'm pretty good around the hoop.",
{"tb": tb},
)
assert len(tb_res) > 0, "Expected non-empty result but got empty."
for r in tb_res:
print(r.model_dump())or if you're open to share your prompt / data model with, I can send over my best guess of a good prompt! We've found these models works even with over 50+ fields / nested and whatnot decently well!
> I can send over my best guess of a good prompt!
Now if you could automate the above process by "fitting" a first draft prompt to a wanted schema, ie where your library makes a few adjustments if some assertions do not pass by have having a chat of its own with the LLM, that would be super useful! Heck i might just implement it myself.
We just need a bit better testing flow within BAML since we do not support adding assertions just yet.