Is there a tool/technique to achieve this? I’m aware that I can use LLMs to do so, or read all pages and find identical text (header/footer), but I want to keep the page number as part of the metadata to ensure better citation on retrieval.
A better approach will be using Textract as it maintains the flow, such as if you have a table going across multiple pages.
Btw, tesseract is not that good in getting accurate data from tables. Use it with caution especially in financial context.
I have made an open source tool to show missing data from tesseract and easy ocr https://github.com/orasik/parsevision/
ColPali is the standard implementation & SOTA. Much better than OCR. We maintain a ready to go retrieval API that implements this: https://github.com/tjmlabs/ColiVara