Hacker News Reader

The problem lies therein that JBIG2 has a compression factor, that is controlled by people who have an incentive to increase it, where few people know what it actually does. And how much testing is really done when is changed? Will the test include text in other scripts, like Chinese or Japanse? It can go wrong in so many ways.

Jpeg artifacts on text are ugly but they tend to stand out. You will see artifacts long before the text becomes unreadable or ambiguous, and that is a good thing.

There are a number of scans I've come across on archive.org where the person uploading has used jbig2 and there are pages where letters like 'e' get swapped for an 's'. My general rule when creating pdfs with scans is: If it's mostly text or line drawings and it will be used in a professional setting use png/flate. If it's a photo use jpg (especially if the source is a jpg).

I'd love to see some examples (especially if the scan appeared to be of good quality and not a grainy 100dpi affair). Do you have any idea what kinds of documents you were browsing when you found them?

https://news.ycombinator.com/item?id=17435514 (found via https://en.wikipedia.org/wiki/DjVu)

There's apparently 22 pages of fluff at the beggining, so you should add that the page numbers. Also "the third line" was meant to be the 3rd line from the bottom.

Direct links (to be opened in DjView):

https://ia800204.us.archive.org/28/items/cu31924026442156/cu...

Thanks! When I convert the jp2k file for that page 58 with ABBYY default ("balanced") jbig2 settings I don't get any of the errors we see in the examples. If I convert to DJVU and purposefully degrade the image to 150DPI, I still don't get the errors on display in this example. Besides the errors talked about in your linked discussion (germauic), the pdf clearly has many 'c' -> 'e' errors which aren't in the jp2k files. Again, I can't reproduce this with the software I use. I've also extensively used djvulibre and you just don't see errors like this from clean scans.

So, I have to conclude that the original conversions were done with very aggressive settings (or very bad jbig2 implementations). If that's they way the internet archive did all its encodings, then I admit there must be several problem documents, and they should re-do the conversions from the original JP2k files they seem to have kept. Not sure if google books made similar mistakes.

There are both lossy and lossless jbig2 variants.

Unless something has changed, Google books has always used the lossy encoding. Quoting the original paper:

    The most obvious way to compress pages is to losslessly encode them as a single 1 bpp image (see Figure 2). However, we get much better compression by using symbol encoding and accepting some loss of image data. Although our compression is lossy it is not clear how much information is actually being lost - the letter forms on a page are obviously supposed to be uniform in shape, the variation comes from printing errors and lack of resolution in image capture.

Probably only a small fraction of errors are JBIG2 substitutions though.

My choice would be the recently standardized JPEG XL designed to replace all of: JPEG, JPEG 2000, PNG, GIF. Among other things, it is supposed to be the first choice for long-term storage. I guess mass adoption will begin after including it in the PDF standard. It is already experimentally in Chrome and Firefox.

2nded

JPEG is optimized for compressing photos, not text or high contrast images.

This only applies to legacy JPEG (1992), not for JPEG XL (2022).

I've been reading some books on my ereader and every now and then I've come across a word that has been misspelled with similar looking characters. now I'm wondering if it's this jbig2 issue