“Some participants have dedicated years of their lives to the program—like Alex Smith, a retiree from Pennsylvania. Over nine years, he transcribed more than 100,000 documents”
Have different LLMs transcribe those same documents and compare to see if the human or machine is or accurate and by how much.
Even if LLMs can push a 99.9% accuracy to 99.99, at least an OCR-based system can be audited. Ask an OCR vendor why the machine confused "Vancouver WA" and "Vancouver CA" and one can get a solid answer based in repeated testing. Ask an LLM vendor why and, at best, you'll get a shrug and some line citing how much better they were in all the other situations.
https://trustdecision.com/resources/blog/revolutionizing-ocr...
> Our internal tests reveal a leap in accuracy from 98.97% to 99.56%, while customer test sets have shown an increase from 95.61% to 98.02%. In some cases where the document photos are unclear or poorly formatted, the accuracy could be improved by over 20% to 30%.
While a small percentage increase, when applied to massive amounts of text it’s a big deal.
Looking at the (admittedly single) example from the National Archives seems a bit more open than perhaps the other two examples. It’s not impossible thst LLMs could help with this
https://trustdecision.com/resources/blog/revolutionizing-ocr...
Insisting we switch to that definition is just being unproductive and unhelpful. And it's pure semantics because you know what they meant.
“Our internal tests reveal a leap in accuracy from 98.97% to 99.56%, while customer test sets have shown an increase from 95.61% to 98.02%. In some cases where the document photos are unclear or poorly formatted, the accuracy could be improved by over 20% to 30%.”
(I always go over the transcript by hand, but I'd have to do that with OCR anyway).
As an example look at subtitle rips for DVD and Blu-ray. The discs store them as images of rendered computer text. A popular format for rippers is SRT, where it will be stored as utf-8 and rendered by the player. So when you rip subtitles, there's an OCR step.
These are computer rendered text in a small handful of fonts. And decent OCR still chokes on it often.
If 100% is your standard, good luck solving anything ever.
Not necessarily, I'd be surprised if I could fully understand my old handwritten notes from when I was in school (years ago), since I've always had messy handwriting and no longer have the context in each subject matter to guess.
LLMs could help in some of those cases, since it would have knowledge of history/chemistry/etc. and could fill in the blanks better than I could at this point. Though the hallucinations would no doubt outweigh it.