Why frontier LLMs can't read the hard documents without experts involved | Hacker News Reader

Why frontier LLMs can't read the hard documents without experts involved

idp-software.com

27 pointschelm12 comments

> We do not sell IDP software and we are not paid by any vendor named here.

The vendor list contains your own product and you, as in the company you co-founded, pretty clearly sells IDP software. https://idp-software.com/vendors/konfuzio/

I can make this more transparent; it's the same issue that Parashift had, which ran https://intelligentdocumentprocessing.com/, which they terminated a month ago.

IDP is not a really sexy market. There are only a few people, who are working in the industry.

I do this in my free time to give small vendors a chance, as big corporates like Rossum, Abbyy, or Kofax (now Tungsten) just rule the market by their ads spent.

I can also make it closed source and ask for a fee to get listed as Gartner would do it in their IDP Magic Quadrant.

I did spend 1/580 of the time on the page konfuzio. Ok, true. And I spent 579/580 on the market. https://idp-software.com/sitemap.xml

Are you high?

168 meters above sea level

a clearly LLM written piece about how frontier models are struggling to get past 76% accuracy on their benchmarks (they call it a "wall") in OCR tasks. that is, feeding it a picture of a document and asking it to extract the text.

The benchmark site is here https://www.idp-leaderboard.org/

They say some specialist models get better results on their benchmarks (Nanonets OCR-3 85.9%)

I linked your board already. You are right.

Do you know a benchmark that tries to measure the bussines accuracy.

Most benchmarks focus on the charackter level.

IDP Software typically uses metadata to map information that is either not readable or missing in the document, e.g. extracting the VAT and mapping the street, house number, cip and city.

I think there are many models and many providers. However, it's really difficult to measure the accuracy on a porcess not just on a character level.

https://idp-software.com/vendors/nanonets/

I saw that the leaderboard is hosted by Nanonets. Totally fine for me. So you might be the expert about Nanonets: Let me know if you want to update your post on my site.

it's not my site, I have nothing to do with nanonets. the information was taken from the article.

tl;dr: years ago, Tesseract was the go to tool to extract text. Nowadays, vLLMs can not only extract the text and the layout but also context and provide structured data or even interpret or map data across documents. Prices dropped significantly, while extraction, classification and modification capabilities increased.

The intelligent document processing (a funny marketing term on top of OCR) market moves from "Can software extract the text", which is normally measured by benchmarks, to can software autonomously run "a" specific company process.

the fallback is called human in the loop, hallucination (LSTM vs. vLLM), prompt engineering.

proof me wrong: the hardest challenge is no longer the OCR accuracy but the integration and issue handling in production. Probably "an agentic team can handle this" ^^

This is rather incoherent.

I mean this is for handwritten OCR.. do humans do better?

I've been using Qwen3.6 to OCR stuff, primary receipts and it frequently accurately reads stuff on mangled/faded/folded documents that I have a hard time with... including handwritten stuff (though that's not flawless).

ahahah, probably not. Looking at my own handwriting: Neither in writing nor in reading.

I find it interesting how the prompt changes the result.

If you let the model focus on the text, the open source got so good in the last year. That's remarkable. When you change to prompt to not only extract the text but also extract specific information, the pure text extraction result gets worse. For me, it worked to run two prompts on the same document to get both in a meaningful accruacy.

I had good luck with using a process where a supervisory pass looked at reasoning traces and improved the instructions. This had major improvements in performance and quality.

I thought about doing a pass with a non-vision-llm and and after having the vision llm give its result show it the non-vision and ask it if it needed to reconsider, but my test input was too easy to it to matter.

One thing I worried about was that my scans are big mostly empty pages, I thought this might not give enough attention to the receipt text and that I should do some preprocessing to crop the image. But, again, the results were too good without doing it that I don't think I could test if its an improvement or not.

I am also a little worried that reasoning about the content might make it hallucinate stuff, and worse have the hallucinations be credible enough to not get caught later. But the improvement it got from reasoning out genuinely unreadable parts made the risk worth it for my use.

Reply on news.ycombinator.com