Hacker News Reader

Comment by MuffinFlavored | Hacker News Reader

> This means you can now use gpt-4-turbo vision to extract structured data from an image!

How consistent and reliable is the extracted structure?

Did they add any kind of concept "for whatever token you generate/think it is next, "unit test it" / make sure it passes some sort of rules?"

View on HN

It's pretty good, but it's not reliable enough to exclude the need to check everything it does.

Same story as basically everything relating to LLMs to be honest.

llama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial to prompt it towards the exact result). So this particular story doesn't need to be the same among all LLMs.

https://github.com/ggerganov/llama.cpp/blob/master/grammars/...

llama.cpp grammars will get you results that definitely conform to your grammar, but that doesn't guarantee that they'll be semantically correct - the model could still hallucinate details incorrectly while returning valid JSON.

Indeed it's not the solution to LLM hallucinations—as far as I know nobody knows a solution to it.

But it is the solution to needing to run the model again and checking the format of the output to ensure that it conforms to you expectations.

^ This.

I've found success generally better with Textract and then passing the OCR'ed text to OpenAI and using Pydantic to get it structured.

In my testing I was better off running the image through AWS Textract then taking the output and feeding it to OpenAI. It was also much cheaper. Of course if all you are looking for is extraction then maybe you don't need OpenAI at all. I used it to clean up the OCR'd data and reformat it.

My experience is that it's pretty good at reading text and pretty bad at understanding layouts. So e.g. asking it to work with tables is asking for trouble.

Yeah it's absolutely horrible at layouts.

I'm not 100% sure it's related but if you ask it to draw bounding boxes around things it's always off by quite a bit.

Even if the GitHub link they just posted above. It made up the year at 2022 and hallucinated the start and end time.

https://github.com/datasette/datasette-extract/issues/19

It's very consistent, check this guy, he was able to structure LLM output using Pydantic in an elegant solution:

https://www.youtube.com/watch?v=yj-wSRJwrrc

> How consistent and reliable is the extracted structure?

That's the $100B question.

Or $T these days

You restrict the model's next prediction to valid JSON tokens. (If you mean format reliability.)

I'm waiting for being able to restrict to a specific JSON Schema

Providing examples generally helps me. Have you checked out the Instructor library too?

Reply on news.ycombinator.com