Hacker News Reader

Comment by yorwba | Hacker News Reader

> we still don't have an easy to use and free high end OCR available

There is Tesseract: https://github.com/tesseract-ocr/tesseract

> Australia's Trove is getting humans to translate them‽

Your link clearly refers to correcting an existing transcription:

"While viewing digitised newspaper and gazette articles, you may notice that the text transcript doesn’t always match the text in the article. You have the power to fix this by editing the transcript to match the article text."

Most likely they used OCR software to generate the initial transcript, but allow users to correct the OCR output because they know the software is not perfect.

View on HN

I used tesseract for something that sounded extremely simple: OCR a date tag and a few characters identifying a large list of photos. The tag is of very high quality, white over black, a 6yo would read it flawlessly. I had to use the "old" tesseract (the new one uses ML), probably because the ML-based one was "inventing" characters or swapping them, and could not reliably identify numbers. Even with the old solution, I had to resize, apply some filters and was barely reaching 99% recognition. For something extremely clean ! So we have ways to go...

Tesseract makes me feel like we've gone backwards in 20 years.

I know there have been improvements, and probably impressive ones, but they are not being implemented in any sort of unison in anything easy to use.

If computers can't even read newspapers, which only have a certain amount of fonts and letters/words to chose from how can we think they can do much else.

Test of - https://trove.nla.gov.au/newspaper/article/151059024

Tesseract starts really well here but then stops after a few lines. No idea why, it just frustrates me.

"FEMALE PIRATE LEADER. HONG KONO, Sunday. After no sign of activity in Bias" - Then Stops.

Trove is below and doesn't get much.

"FEMALE PIRATE LEADER. HOHCrKOiro. SanOV. After twiigiiJofswUTlty In »» Xay far ? jux. pirates, lei br a woman, aged 28, EagUA tpio^K, attacked the JspaBMe steaiktr TM1 Slant ? betwsan jbaojr and"

ABBYY Finereader is OK -

"FEMALE PIRATE LEADER BOKO KOHO, Sunday, After no ties of activity la Bin Bay for a year. pirates. lad by a woman, aged 28. English speaking attacked the Japanese steamer"

Tesseract is great in certain situations but a lot comes down to having a robust preprocessing pipeline and a correction flow. I wrote about this a few years back:

https://chris.improbable.org/2014/3/17/content-search-on-a-b...

https://blogs.loc.gov/thesignal/2014/08/making-scanned-conte...

Basically the big problems are getting the content deskewed (even a slight rotation will cause accuracy to plummet, which is a problem if there was page curl or a flaw in the original printing process), breaking text into clean segments (non-trivial in newspaper layouts), and dealing with noise from dust or content from the other side of the page bleeding through. The collection I’ve worked the most with (https://chroniclingamerica.loc.gov/) also had a lot of problems due to many collections having been scanned from microfilm first. Tesseract 4 is better but in my testing you aren’t going to see revolutionary improvements without investing in tooling to identify segments and clean them up before passing them to Tesseract.

Since that entire collection is public domain and freely available for download (https://chroniclingamerica.loc.gov/data/ or s3://ndnp-batches), researchers have used various ML tools on it and that definitely looks promising but is not a silver bullet by any means. There are some trained files available here along with a large public S3 dataset:

https://news-navigator.labs.loc.gov/

Reply on news.ycombinator.com