Hacker News Reader

Comment by simonw | Hacker News Reader

They also added both JSON and function support to the vision model - previously it didn't have those.

This means you can now use gpt-4-turbo vision to extract structured data from an image!

I was previously using a nasty hack where I'd run the image through the vision model to extract just the text, then run that text through regular gpt-4-turbo to extract structured data. I ditched that hack just now: https://github.com/datasette/datasette-extract/issues/19

View on HN

One of the OpenAI PM's was also saying the model got substantially better at math: https://x.com/owencm/status/1777770827985150022

I'm trying it for coding and have added it to my VS Code Copilot extension Overall I'd say it's better at coding than the previous GPT-4 Turbo. https://double.bot if anyone wants to try it :)

Is being good math that important to the ChatGPT users, though? The ChatGPT's ability to do math is so limited that I'm not sure what math problems we have to ask ChatGPT to solve.

It means you don’t have to be as sketched out if you’re looking for something that requires basic math. Imagine generating the correct result of a unit test or something. I wouldn’t trust it either way, but I think this is a believable example.

Math is not just arithmetic and yes better math does help at least some GPT-4 users.

Do you have examples? The other commenter's example of generating correct unit test results is indeed interesting, and I was wondering if there are other cases.

Not sure what you are looking for, but perhaps these thoughts can help. Better geometry understanding helps with spatial tasks. Better probability theory, discrete math, and linear algebra background helps with algo development. More broadly, being better at things that a mathematician is good at, could help tackle complicated tasks in finance, science, or engineering.

We've been using `gpt-4-1106-vision-preview` and simply prompting the model to return json, with excellent results: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress).

> This means you can now use gpt-4-turbo vision to extract structured data from an image!

How consistent and reliable is the extracted structure?

Did they add any kind of concept "for whatever token you generate/think it is next, "unit test it" / make sure it passes some sort of rules?"

My experience is that it's pretty good at reading text and pretty bad at understanding layouts. So e.g. asking it to work with tables is asking for trouble.

Yeah it's absolutely horrible at layouts.

I'm not 100% sure it's related but if you ask it to draw bounding boxes around things it's always off by quite a bit.

You restrict the model's next prediction to valid JSON tokens. (If you mean format reliability.)

I'm waiting for being able to restrict to a specific JSON Schema

Providing examples generally helps me. Have you checked out the Instructor library too?

> How consistent and reliable is the extracted structure?

That's the $100B question.

Or $T these days

It's pretty good, but it's not reliable enough to exclude the need to check everything it does.

Same story as basically everything relating to LLMs to be honest.

^ This.

I've found success generally better with Textract and then passing the OCR'ed text to OpenAI and using Pydantic to get it structured.

llama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial to prompt it towards the exact result). So this particular story doesn't need to be the same among all LLMs.

https://github.com/ggerganov/llama.cpp/blob/master/grammars/...

llama.cpp grammars will get you results that definitely conform to your grammar, but that doesn't guarantee that they'll be semantically correct - the model could still hallucinate details incorrectly while returning valid JSON.

Indeed it's not the solution to LLM hallucinations—as far as I know nobody knows a solution to it.

But it is the solution to needing to run the model again and checking the format of the output to ensure that it conforms to you expectations.

In my testing I was better off running the image through AWS Textract then taking the output and feeding it to OpenAI. It was also much cheaper. Of course if all you are looking for is extraction then maybe you don't need OpenAI at all. I used it to clean up the OCR'd data and reformat it.

It's very consistent, check this guy, he was able to structure LLM output using Pydantic in an elegant solution:

https://www.youtube.com/watch?v=yj-wSRJwrrc

Even if the GitHub link they just posted above. It made up the year at 2022 and hallucinated the start and end time.

https://github.com/datasette/datasette-extract/issues/19

Great work, thank you, Simon.

Have you noticed GPT-4 Vision do weird selections of dates for 2019? In doing some processing of some data for work, I ended up switching to textract and then passing the OCR'ed text to GPT-4 and mapping to a Pydantic schema due to this issue.

He, cool to hear you're doing something like this too! We ended up close, but we also need good spatial relationships, which GPT4V isn't great at, so we're using other OCR system and adding the result to the context.

Does anyone know if we can access the same on Azure OpenAI? It still shows vision-preview at my end in West US region with no JSON mode

What time is the event? I think 18:00 - 22:00 is completely fabricated.

Reply on news.ycombinator.com