Unified Line and Paragraph Detection by Graph Convolutional Networks (2022) | Hacker News Reader

Unified Line and Paragraph Detection by Graph Convolutional Networks (2022)

arxiv.org

Interesting. Two engineers at Apple worked on something similar that would slurp character bounding boxes from a PDF page and reconstruct paragraphs, columns, tables, etc.

It was surfaced in iOS a decade ago as "tap to zoom" feature for PDFs. It's funny — as with a lot of things there was a lot of sophisticated engineering under the hood and then marketing simply wants it to detect a tap in a paragraph and zoom to its bounds.

I can't think of the last time I read a PDF on my phone or I would test it to see if it still works as I remember.

Wow, I had no idea. That’s pretty cool. I don’t read a ton of pdfs on my phone, but I always find it super frustrating when that functionality is missing somewhere else.

I think we're thinking the same thing, but this is one of those little features on modern smartphones that we used to take for granted. Being table to tap to zoom on a document and have it snap to the right spot was a big feature improvement, now we expect it.

One of those engineers came up with idea of using cluster analysis for word grouping. The other engineer realized that we had a k-cluster problem in the general case, but needed to figure out what k was. (In the simplest case it is 2 - letter gaps and word gaps). They used the differences between the anchor points of glyphs.

I can confirm that tap to zoom and word higlight still works on many PDFs. PDF documents are so varied in structure that it is impossible to say that any technique always works.

Not just PDFs, you can also do the same thing (double tap to zoom to a paragraph) in Safari on any web page.

I was there when we added it to Preview ;) I think Nils wrote the code to hook it up, because he also worked on the loupe and how it would automatically lock onto the bounding box of a line.

Hi, Matt. Yeah, one of the engineers is on HN from time to time — I thought he might see the post and respond. (Your reminding me of Nils made me smile this morning.)

Ahem. Actually what I am most proud of is that I made pinch to zoom keep characters focussed during zooming. At that time, Safari let things get blurry until the user stopped zooming and then it snapped to focus. That is what I was supposed to do, but I decided to use mipmaps instead.

Ha ha, no idea from your username.

I often find myself making a screenshot and copy pasting from the image because selection in many apps is so broken. Especially for PDF, the iOS viewer has completely useless selection.

Should there be a million-dollar prize for developing a way to parse and reflow the text in PDFs in real time? Reading PDFs on screens is a huge headache.

I think there already might be implicitly, I think you could sell that as a product.

Various apps already do this, if you find yourself reading lots of PDFs on phones. For example I use PDF Expert on iOS to do this. It’s not perfect—depending on the quality of the PDF there might be weird artifacts in the reflowed text (e.g., “ff” ligatures getting mapped to the Unicode ligature character “Latin Small Ligature Ff (U+FB00)” which breaks copy/paste/search).

But for PDFs which are really hard to read on a phone otherwise, it’s really a nice investment.

An interesting problem, particularly with handwritten documents. And not limited to what they worked on, from their conclusion: "...the number of clustering levels is not limited to two, since document layout tasks are extremely diverse in nature. Paragraphs can further be clustered into text columns or sections, which may belong to even higher level blocks. Figure 16 shows a physical paragraph, or part of a semantic paragraph which spans across multiple text columns..." Not to mention text wrapping around figures.

The general field is called "document structure analysis" or "document layout analysis." There's been lots of work--at a cursory glance at this article, I'm not sure they've discussed that literature.

I worked on a similar problem a decade or so, although our work was done mostly by hand. We were trying to not only read in (bilingual) dictionaries using OCR, but turn them into dictionary entries, and then parse each entry into its parts (headword, part of speech, definitions or glosses, example sentences, subentries...). I won't go into details, but to our surprise one of the most difficult parts for the machine to get right was recognizing bold or italicized text.

See (as one of several patents about this): https://ppubs.uspto.gov/api/patents/html/7899826?source=USPA...

Thanks, but I get "unauthorized" with this URL. I tried cutting off everything from the '?' right, but still got the same message.

https://ppubs.uspto.gov/api/pdf/downloadPdf/8166037?requestT...

Only two weeks ago, I was trying to save an online-only book of 24 chapters. The text is filled with images to help illustrate and contextualize the content. I individually saved each chapter as PDFs and ran a few different command line tools to try to extract the contents to plain text. They all came out badly disjointed. Even tools that were meant to do what this paper describes failed miserably at reconstructing naturally flowing text.

While this isn't something I need on a regular basis, it's timely news to hear about someone making progress on what seems like it ought to be a straightforward problem to solve. As the results of my efforts show, it must not be nearly as simple as one might expect.

You could try splitting the book into 1 page PDFs. Send to Gemini flash 2.5 and ask it to OCR to markdown format. It's about USD $0.006/page. It works well for one of my clients.

This worked perfectly:

https://news.ycombinator.com/item?id=45443719

Docling from IBM and Markitdown from Microsoft are reasonably reliable if you didn't try them also take the extra step to get image summaries in plain text from a VLM it's useful of you want to feed final results to an LLM later. Or first try to skip all that with jina.reader or firecrawl llmstxt they will extract directly from the website so simple but sometimes it works sometimes it doesn't.

Dupe: https://news.ycombinator.com/item?id=45323027

HN title doesn't match the link or the comments? Dang?

2022, and we need this in browser reader modes.

remotely related, but I have yet to find a solution for page classification in a document for tables, i.e. a classifier that returns the index of pages containing tables in a document that is reliable

solutions using things like img2table or pymupdf are really bad (pymupdf is not even reliable for text pdfs)

In my experience, this task is incredibly difficult for generality.

Handcrafting based on the dataset is the only way to get high performance.

If you are a fan of irony, the paper is also available in PDF format.

It's like rai-ain, on your wedding day.

Would this be of help with YouTubes auto-generated subtitles? I hate the fact the words appear just in time one by one, unlike traditional complete sentences that appear fully one at a time.

Reply on news.ycombinator.com