Hacker News Reader

Comment by jacquesm | Hacker News Reader

I've been reading quite a few DjVu encoded documents and have yet to see a single occurrence of what you describe. If there are such errors they will be visually indistinguishable from the original, and a good OCR package would likely identify the right letter given the context.

Do you have some concrete examples of corrupted texts? I'd very much like them to use as test cases in a document processing pipeline.

View on HN

This document caused me a fair bit of consternation transcribing it on a site that prefers DjVu files to PDF:

DjVU: https://ia800204.us.archive.org/28/items/cu31924026442156/cu...

PDF: https://ia800204.us.archive.org/28/items/cu31924026442156/cu...

I don't have an easy way to find all of the errors I observed but typical ones are on page 58 at the end of the third line, 'Germanic' with a little bit of bleed between the serifs at the bottom of 'n' gets transformed into 'Germauic', and on page 420 at the end of line 7, 'compass' with a small dot in the middle of the 'o' becomes 'cempass'.

FWIW I've transcribed a bunch of other DjVu files produced by the Internet Archive and not noticed anything like this.

Excellent. Thank you very much for providing this, it will make my life a lot easier (or, initially: a lot harder :) ).

If I can reliably flag problem cases then at least I will know which files are going to have a lot of manual work.

have yet to see a single occurrence of what you describe

How would you know? Maybe if a document had an unusual misspelling. But how would you know if DjVu munged a 3 into an 8?

But this is an entirely hypothetical weakness. Without at least one example of it happening in the wild, it may as well not exist.

Let me assure you, this does exist in the wild. I spotted it many times, especially in documents scanned 15+ years ago, when space and bandwidth were an issue and people did anything to reduce file size, at the cost of quality. It was rare enough to not make the format unusable, but not just once, making you worry "was it 3 sp. or 8 sp.?"

It seems also that the problem happens more often with cyrillic glyphs which have less descending/accending elements that typical latin ones.

With DjVu or with ocr'd documents in general?

While of course it's an issue with any scanned docs, the DjVu compression method makes it harsher, as, say, you can end with _all_ '8's in document replaced with '3' (or, say, 'h's with 'n's), at the same time keeping seemingly decent appearance so that you don't suspect that something is wrong.

I'd love to see some concrete examples.

I totally understand how this could theoretically happen but it all hinges on it actually happening in the documents that I'm working with and to have a test case where it happens for sure would make all the difference in determining whether or not the documents I'm working with are at risk or not and if so if I can detect which ones are at risk (that would be half the battle won).

It would have to be a case where a human would see a 3 or an 8 (or a similar transposition with other glyphs) resulting in a document that is corrupted in such a way that afterwards the human would see the (invalid) alternative.

Even a single instance of this happening would be very relevant.

Well I know that there are examples in some of 8k+ djvu files on my hard disk (mostly old textbooks), as I learned about this problem many years ago from live experience. But I couldn't invent a method to find an example deliberately, so you can only take my word that they exist.

To clarify, it's not about OCR errors, where advanced OCR engine could use dictionary, language autorecognition etc. It's about dvju-compressed text scans without OCR layer. The compression method relies only on glyphs similarity. The compression algorithm could not only mistake '3' as '8' with small gaps left, it would replace this dirty '3' with the image of reference '8' image, so that human reader think that he sees a scan, i.e. an image of page scanned, while in fact it sees 'edited' image, not corresponding to actual page.

> so you can only take my word that they exist.

I take your word, there is a technical reason behind this, not that I don't believe you.

> To clarify, it's not about OCR errors, where advanced OCR engine could use dictionary, language autorecognition etc. It's about dvju-compressed text scans without OCR layer. The compression method relies only on glyphs similarity.

Yes, I totally get that. It's about DjVu's compressor replacing the image of one character with the image of another.

IIRC Xerox revoked lots of their copiers which used the same compression algorithm internally after this was demonstrated by a document with lots of numbers. Google it, it's not that hard to find (I'm on a phone now so it's not very convenient for me atm)

> Let me assure you, this does exist in the wild.

You would obtain more credibility by actually providing such an example.

> But how would you know if DjVu munged a 3 into an 8?

By comparing it with the original, surely.

reaperducer's point was that jacquesm almost certainly hasn't done so, and so can't be entirely confident of not having been the victim of corrupt text.

In my case I don't actually have the originals so it would be hard to impossible to verify on this dataset but if there are concrete examples of such errors then I could use that to identify documents that are more at risk of such errors.

Reply on news.ycombinator.com