What's so hard about PDF text extraction? | Hacker News Reader

What's so hard about PDF text extraction?

filingdb.com

11 pointsdevy15 comments

This is a humbling read. Rn my company is trying to figure this out for PDFs in our domain. Any recommendations for saas solutions or libraries that we ought to explore?

(( if your answer is no because this is part of your secret sauce, I totally understand ))

I've done a fair bit of it in the past - the issues with missing spaces or substitute characters used for spaces certainly rings a bell. I know quite a few libraries in the .NET space for processing PDFs, including at least one open-source one (PDFSharp from memory) which we had in our source code repo, and made up something like 75% of the total LOC for our product, despite PDF processing being a relatively minor feature). We ended up abandoning that though, can't quite remember why.

The real question is how did such a torturous (once proprietary) format manage to become the only ubiquitous standard for paginated, self-contained, formatted text suitable for printing. In principle HTML could actually do the job, but arguably it's not the ideal tool either, and I've not seen an example of an HTML document that just prints as expected across multiple pages either.

Odd that this got posted almost the same time as https://news.ycombinator.com/item?id=33145498 btw...

Yes, I saw the other submission and post this (as I read it a few months back).

It's a hard enough domain area that warrants a startup to solve the pain point.

Pdf is meant for document reproducability. The fact it's glyph soup impedes tampering, which could be considered a feature by those who don't want their documents tampered with. i.e. to be presented to others exactly how they were when created.

If you want an editable document, use a word document instead.

HTML does have page-break-before and page-break-after css properties, but css could do with a "scale element to fit page", "only this element on a page" and "center on page" css properties. Even the print specific css properties that do exist currently are not well supported by chrome. Google really needs to fix that situation IMHO.

It's a pretty terrible way to prevent tampering though - unless it has a digital signature (and all the necessary proof of signature authenticity via certificate authorities etc. etc.). Yet your average punter is far more likely to believe a PDF hasn't been tampered with than even a webpage hosted on a TLS-protected website. Word documents are not just proprietary* but for a high percentage of users by default open up in an editor designed precisely for "tampering", and certainly not for reading. There's definitely a need for a document format that serves the purpose PDF does now, I just wish PDF wasn't that format.

* sure, docx may be a form of OpenXML but it's still largely based around MS Word's proprietary behaviour - very few other tools can reliably display all docx files well.

pdfs should certainly have more robust means to prevent tampering I agree. pdf integrity assurances aren't much better than ROT13 is for cryptography.

This is what PDF digital signatures are for. They allow you to make sure that the PDF hasn't been tempered with. However, that is not there original intention.

From what I remember the PDF standards groups are currently investigating adding something just for temper detection.

Being able to place a digital signature on PDF as proof that you've read it and, for contracts, e.g. as a way of indicating your acceptance of its terms etc. would be far better than the rather absurd method used now where you can just type in your name in some fancy cursive looking font and send it back as supposed proof of your acknowledgement.

The problem with digital signatures is that you need to have an appropriate certificate to do this. If the documents are only used internally, then some self-signed certificates or a company-internal certificate authority would be fine. But usually you want an official certificate.

And many certificate authorities sell such certificates but for high prices.

Typing in the name or signing with a touch screen is actually considered a valid and binding, official signature in most countries. It is the lowest form of a signature equivalent to a written signature.

Can you use LetsEncrypt or similar free services for certificates for PDF signing? Signing using a touch screen/ mouse/scanned-in written signature at least provides some small element of protection against forgery and proof of acknowledgment - typing gives you nothing really. And yet it's what I mostly end up doing because the alternatives are too cumbersome.

I don't think LetsEncrypt provides such a service. Additionally, the certificate must/should come from a CA that is included in the AATL (https://helpx.adobe.com/acrobat/kb/approved-trust-list1.html), otherwise you won't get the nice visual check mark that the signature is okay on Adobe Reader (and, let's face it, this is what most people are probably gonna be using on Windows).

Agreed on cumbersome, I have a PNG file with transparency containing my signature and use Xournal++ (https://xournalpp.github.io/) for placing it on the PDFs. That works fine but still takes more time than I want to spend. But it's faster than typing it out every time.

If the PDF creation software supports tagged PDF, text extraction gets much easier because all the text and structural information is preserved. This also allows "reflowing" of contents, similar to how a HTML page reflows on smaller screens.

Extracting text from a pdf is akin to using text-to-speech software to produce a .wav file then expecting to still be able to edit the original text when only having the .wav file.

Anyone wanting to extract text from a pdf would do better to hand it to a pool of typists for them to retype into a new document.

Or better yet, edit the original document that the pdf was generated from.

I find that most PDFs extract paragraphs out of sequence. It's so annoying.

My most effective (so far) extractive program is 'pdftotext' and using the '-layout' option:

         pdftotext -layout my_file.pdf

Stuffing keywords outside the page's bounding box seem like a good way to bypass (dumb) resume scanners

Reply on news.ycombinator.com