Hacker News Reader

Comment by moralestapia | Hacker News Reader

>From a machine-parsing perspective PDF files are a nightmare.

Can someone with experience care to explain why? Does it have to do with each letter having an absolute position on the document? I have no clue, to be honest.

View on HN

You have to essentially render the document yourself in order to figure out what the order of chunks is. Then you might be able to extract content from the chunks you're interested in - or not. A given zipped chunk might be literally anything.

Didier Stevens is the best expert of PDFs that I know of: https://blog.didierstevens.com/programs/pdf-tools/

There's a whole pile of different gotchas.

* identifying characters that won't actually print [white text; zero-size font; beneath another layer; not in printable space]. Once this lead to every letter aappppeeaarriinngg ttwwiiccee..

* text in fonts where a != aa [leaving the text as a complicated substitution cypher; caused by font embedding for only the characters in the document]

* text in images

* no spaces: have to infer from gaps in letters

And these are generated by a whole host of different software with different assumptions, and you never know if there's something else you're missing.

I have no experience in handling raw PDF data, but as a user, I sometimes notice that the computer is not reading the PDF text the same way as I'm reading it. Here are a few examples:

1. When searching (Ctrl+F) a commonly used phrase that occurs multiple times in a PDF, some occurrences fail to show up because of line breaks, accidental hyphenation, etc.

2. Once in a while, I come across PDF files where searches for words containing "fi", "ff", etc. fail because of some insane ligature glyph replacements.

3. Some PDF files that have a two-column layout for text still treat lines across the two columns as one line. Search fails again.

Yeah, pretty much exactly what you said. Since PDF is focused on presentation rather than content it can write text content in any order, and the rules for converting the byte values to unicode values are extremely complex, supporting many different font formats. Some fonts (type 3) don't even include mappings to unicode in some scenarios, instead just encoding the appearance of glyphs and not their meaning.

Assuming you have a reasonable PDF file you have to parse the entire page content stream which includes things like colors, lines, Bezier curves, etc, extract the text showing operations and then stitch the letters back in to words and then words back into reading order, as best you can.

Many sensible PDF producers encode letters and whitespaces reasonably thereby preserving reading order, but this is far from universal.

For an idea of what content stream parsing involves, this is how I currently do it: https://github.com/UglyToad/PdfPig/blob/master/src/UglyToad....

It was never really designed for that, only for display and printing. There are some features that let you mark up the text for easier searching and selection, not every producer will use them though.

It is a bit of a dogs dinner now, mostly because of backwards compatibility. XPS is better but obviously failed in the market.

Reply on news.ycombinator.com