Hacker News Reader

It's pretty good, but it's not reliable enough to exclude the need to check everything it does.

Same story as basically everything relating to LLMs to be honest.

llama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial to prompt it towards the exact result). So this particular story doesn't need to be the same among all LLMs.

https://github.com/ggerganov/llama.cpp/blob/master/grammars/...

llama.cpp grammars will get you results that definitely conform to your grammar, but that doesn't guarantee that they'll be semantically correct - the model could still hallucinate details incorrectly while returning valid JSON.

Indeed it's not the solution to LLM hallucinations—as far as I know nobody knows a solution to it.

But it is the solution to needing to run the model again and checking the format of the output to ensure that it conforms to you expectations.

^ This.

I've found success generally better with Textract and then passing the OCR'ed text to OpenAI and using Pydantic to get it structured.

Reply on news.ycombinator.com