Comment by serjester | Hacker News Reader

Comment by serjester | Hacker News Reader

It should be noted for some reason OpenAI prices GPT-4o-mini image requests at the same price as GPT-4o. I have a similar library but we found OpenAI has subtle OCR inconsistencies with tables (numbers will be inaccurate). Gemini Flash, for all its faults, seems to do really well as a replacement while being significantly cheaper.

Here’s our pricing comparison:

*Gemini Pro* - $0.66 per 1k image inputs (batch) - $1.88 per text output (batch API, 1k tokens) - 395 pages per dollar

*Gemini Flash* - $0.066 per 1k images (batch) - $0.53 per text output (batch API, 1k tokens) - 1693 pages per dollar

*GPT-4o* - $1.91 per 1k images (batch) - $3.75 per text output (batch API, 1k tokens) - 177 pages per dollar

*GPT-4o-mini* - $1.91 per 1k images (batch) - $0.30 per text output (batch API, 1k tokens) - 452 pages per dollar

[1] https://community.openai.com/t/super-high-token-usage-with-g...

[2] https://github.com/Filimoa/open-parse

Interesting. It didn't seem like gpt-4o-mini was priced the same as gpt-4o during our testing. We're relying on OpenAI usage page of course, which doesn't give as much request by request pricing. But we didn't see any huge usage spike after testing all weekend.

For our testing we ran a 1000 page document set, all treated as images. We got to about 25M input / 0.4M output tokens for 1000 pages. Which would be a pretty noticeable difference based on the listed token prices.

gpt-4o-mini => (24M/1M * $0.15) + (0.4M/1M * 0.60) = $4.10

gpt-4o => (24M/1M * $5.00) + (0.4M/1M * 15.00) = $126.00

The pricing is strange because the same images will use up 30X more tokens with mini. They even show this in the pricing calculator.

[1] https://openai.com/api/pricing/

Indeed it does. But also the price for output tokens of the OCR is cheaper. So in total it's still much cheaper with gpt-4o-mini.

That price compares favourably with AWS Textract. Has anyone compared their performance? Because a recent post about OCR had Textract at or near the top in terms of quality.

I’m using AWS textract for scanning grocery receipts and i find it does it very well and fast. Can you say which performance metric you have in mind?

Can you locate that post? In my own experience, Google Document AI has superior quality but I'm looking for something a bit more objective and scientific.

My testing was admitedly limited (but focused on cases where GPT-4o was struggling), but in my experience Claude 3.5 also produced fewer hallucinations than GPT when doing OCR with figures / tables recognition. It is of course quite pricey as well.

However both GPT and Claude still struggle with documents that are very heavy on infographics, figures and tables.

Reply on news.ycombinator.com