Hacker News Reader

Comment by rayiner | Hacker News Reader

Is coding "solved?" I've been using Fable and Grok to vibecode a library to improve text extraction for legal PDFs. (Which have things like line numbers in transcripts that messes up text extraction.) This is a simple, dumb problem that should be like killing a mouse with a nuke. But I got to "pretty good" with a few iterations of not looking at the code or design. The program works, pretty well. But the design and heuristics are only 80% of the way there. Now, the models are just "churning" on the existing design, making ad hoc changes struggling to get to the last 20%. It seems like I'm going to have to turn my brain back on and think about what I actually want to do, which I was trying to avoid.

It's better than writing assembly! But it still seems to be in the realm of "a new kind of tool" even for the actual coding part, without even caring about what the code looks like or security or anything like that.

View on HN

Without knowing details of your approach, I would venture that your challenge is not with the coding per se but extracting structured data from PDFs. It’s a surprisingly hard problem because PDFs are optimized for preserving the visual structure and layout of the content for precise rendering and printing… NOT for preserving the logical structure of the data!

Which is why the best results these days for extracting structured data from PDFs is by having the model do it directly rather than writing code to do it. It literally takes that level of intelligence to be reliable at it.

A common approach is to provide the model with a template or structured schema describing the format you want the data in, and the PDF itself, and it should return a JSON with the appropriate values filled in. It won’t be 100% but probably higher than what you’re seeing now.

The use case is basically the same ones where a programmer would use grep on source files, and it’s too slow to run the model on it directly for the same reasons.

Existing tools do text extraction (pdftotext) but not that great when there’s internal numbering and stuff like that. It has to be a heuristic approach, and it won’t be perfect. But it’s an interesting data point that the model isn’t able to figure out the best heuristic itself. Or even make incremental progress towards one after a certain point.

I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from pdftotext yourself?

My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.

A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middle of line numbers. If you interpret the document as having a single line, you'll get numbers mixed up with the text, which can throw off efforts to find particular phrases. PDF builders also insert all sorts of weird crap into the OCR layers that has to get normalized out.

It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.

Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!)

As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.

Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!

> This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.

Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!

IIRC Claude reads PDFs natively, or as images. One page is 3k tokens IIRC, so you can get 100 pages into context and still have plenty of space to reason or work around that. Only for longer and text-specific documents it uses pdftotext, but that's optional.

>It's just a pretty pedestrian data-munging problem

In other words, how complicated could it be?

Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.

Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff.

Do you use a public set of documents? I bet I could almost oneshot this with my harness :p

Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions.

Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-i...

The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.

Here’s the oneshot, not sure if it’s slop or not though :)

https://kolinko.eu/pdf-reading-order/

But I wonder about your opinion.

I had Claude evaluate my vibe-coded extractor against yours on the EcoFactor corpus, which I didn't test on, and an internal corpus I have:

> The EcoFactor result inverts. There, on hand-authored gold over 21 pages of clean native text, pdfgeo won the accuracy checks 25/42 to 19/42 — its geometry work on multi-column patents, table cells and timestamp lanes is genuinely better, and nothing here contradicts that. But this corpus is 21,652 pages of mostly scanned, OCR'd, skewed exhibits, and at that scale pdfgeo's robustness is the binding constraint: it loses 6.4% of the corpus to two trivial crashes and an over-strict rotation test, duplicates text on 34 files, and drops 80% of wrap hyphens. lawpdf extracted every page, 7× faster, with 2.5× less furniture leakage.

It looks like the actual geometric analysis came out the same general shape, which is an ad hoc heuristic with hard-coded values attempting to figure out where the gutter is located, etc. Seems like both have some overfitting to the text data set.

If you have more test data I can try this with my harness - you can email me at kolinko@gmail.com btw :)

Thanks for the benchmark, I was seriously looking forward to it! Would you consider such even results good for a one-shot? I wonder how well my harness performs :)

What harness? And why not just Fable+Opus?

Personally I use only one family for code editing, other families for code navigation.

Models tend to reason about the code using their own intuition and putting other families onto the same codebases may end up with them getting confused. Esp if that other model is as unhinged as Grok.

Also, harness matters a ton. Use a wrong harness with a good model and you’ll get terrible results.

Reply on news.ycombinator.com