> accuracy is measured with the Needleman-Wunsch algorithm

> Crucially, we’ve seen very few instances where specific numerical values are actually misread. This suggests that most of Gemini’s “errors” are superficial formatting choices rather than substantive inaccuracies. We attach examples of these failure cases below [1].

> Beyond table parsing, Gemini consistently delivers near-perfect accuracy across all other facets of PDF-to-markdown conversion.

That seems fairly useful to me, no? Maybe not for mission critical applications, but for a lot of use cases, this seems to be good enough. I'm excited to try these prompts on my own later.

This is "good enough" for Banks to use when doing due diligence. You'd be surprised how much noise is in the system with the current state of the art: algorithms/web scrapers and entire buildings of humans in places like India.

would encourage you to take a look at some of the real data here! https://huggingface.co/spaces/reducto/rd_table_bench

you'll find that most of the errors here are structural issues with the table or inability to parse some special characters. tables can get crazy!

Author here — measuring accuracy in table parsing is surprisingly challenging. Subtle, almost imperceptible differences in how a table is parsed may not affect the reader's understanding but can significantly impact benchmark performance. For all practical purposes, I'd say it's near perfect (also keep in mind the benchmark is on very challenging tables).

I guess 90% is for "benchmark", which is typically tailored to be challenging to parse.

having seen some of these tables, I would guess that's probably above a layperson's score . Some are very complicated or just misleadingly structured.

Switching from manual data entry to approval