RAGFlow is an open-source RAG engine based on OCR and document parsing

What's the name of the layout recorgniser model? I did not have a good experience extracting layout from tables, especially those without column boundaries (space instead of lines to demarcate boundaries)

This is because PDF has so many different versions. A third-party tools like pdfplumber won't fit it all. For example, using pdfplumber to parse some PDFs will cause the system to raise exceptions. Sometimes fitz works in situations where pdfplumber won't handle well. It looks a bit complicated, but RAGFlow is using multiple parsing tools to handle different types of PDFs.

It's trivial to run a proxy server that routes all OpenAi calls to another LLM, even local ones. See litellm-proxy.

I see a `LocalLLM` chat model where it looks like you can pass a host/port (for example, ollama's)

Looks like they do but aren't really documented yet:

https://github.com/infiniflow/ragflow/pull/119

RAGFlow will support more LLMs, including locally deployed LLMs.

Do you have any recommendations for extraction from Powerpoint documents? Those seem like the worst since each of the layouts tend to be unique (unlike the AMEX Statements in the example)

Cool! This is really helpful.

Actually we've tried almost all lof existing open source models for document processing, and none of them performs well for complex documents, especially those having complicated tables, such as tables cells without borders, cells need to be combined,...,etc. Although adopting LLMs to perform such document understanding tasks is more scalable, it requires much more data and computation power to achieve similar results. That's why we design such models start from scratch.

It is open-source though. Just rip it off and make that PDF() class.

Each project has its own detailed requirements and scenarios, and we cannot demand that each project use same library to implement similar functions

Looks like they do but aren't really documented yet:

https://github.com/infiniflow/ragflow/pull/119

To be honest, RAGFlow already supports this but has not documented this local deployment process yet, as we are still working on simplifying this process, and will release this feature soon. Please keep tuned!

Hi bschmidt1, This is a good feature. We do plan to support it soon. Please stay tuned. If you have further suggesions, welcome to file an issue with us.

Thanks for your nice suggestion. We train the model using YOLO, but during inference, the model is converted into ONNX and we use ONNXRuntime for the model inference. As a result, YOLO itself is not included in the software package. We will open the training code in the repo soon.

Hello friend,

I would like to recommend the FigJam feature in Figma to you, a common tool used by high-tech companies for creating Information Architecture (IA). Our company always adheres to the tool usage standards of high-tech companies.

For guidance on how to use the FigJam feature in Figma, you can refer to this link: https://youtu.be/axDzyLEfYgU?si=V6tqO_tEUKYuLxrL (or search for FigJam on YouTube).

Here are three quick tips on how to efficiently create Information Architecture（Also called System Architecture）:

Start with the Sketch method by drawing your product's Information architecture on paper. Communicate with your PM, design, and development team to validate its effectiveness.

Open FigJam in Figma to turn it into an electronic version. This step is crucial as design and development will follow the version in FigJam for collaborative work.

When creating your Information Architecture (IA), first identify the core functionalities of your product, represented by one color. Next, determine what sub-functions each core functionality can be divided into, marked by a second color. Finally, decide what detailed functionalities compose each sub-function, indicated by a third color. Continue in this manner until you complete the entire IA construction.

It’s important to note that perfecting the IA is not a linear process; it will go through multiple iterations and modifications. Every great product undergoes this process. Also, initially, you can focus on creating an IA for just one core functional module (usually the innovative feature with the highest user pain point) without defining the entire scope. In essence, flexibly establishing the IA to achieve company goals is the primary task.

Ok we've taken deep document understanding out of the title above. Thanks!

I am curious about the performance of their OCR and layout and table detection. Hopefully it’s on par with Amazon, Google, or Microsoft’s tools.

A bit off topic but every time I see DTrOCR I remember that marketing is a good idea because DTrOCR[1] and Fuyu[2] are basically the same architecture[3].

[1] https://arxiv.org/pdf/2308.15996v1.pdf [2] https://www.adept.ai/blog/fuyu-8b [3] If you don't want to search for the figures I made a tiny post about it on my weblog: https://weblog.snats.xyz/posts/2024/02/16/

RAGFlow uses Yolov8 for its OCR/layout recognition/TSR(table structure recognition). And RAGFlow uses large amount private data to train these models for them to perform well in some specialized scenarios.

We've used YOLOv8 as the object detection model, and use some public datasets, such as PubTable, CDLA, together with some private data to train the model. The model on Huggingface is the one trained using public dataset, and we would open this work later. We use YOLOv8 just because we want to let the document parser run without GPU, I think you could also try any other object detection models such as Detectron, and use the public datasets to train the model as well. We've not used transfomers for this task, because given limited data, it could not outperform traditional CNN based models.

The authors are not native English speakers (and what you’re seeing in the name is not something most non native English speakers would spot).

I understand it’s funny to point and laugh “ha ha dumb dudes didn’t think about something female related because they’re dudes”, but it’s worth remembering that not everyone is from the anglosphere.

Also you might find this normal but I cannot imagine my female colleagues being amused at me calling one of them up asking to “consult” about a name solely because she’s a woman.

The assumption that author didn't talk to a woman sounds pretty knee-jerk to me. Can you explain what you mean?

"On the rag" is a semi-common slang for menstruation, but "rag" has way more meanings than just menstrual pads. IMO, as a marketing term, "RAGflow" sounds about as odd as "Nintendo Wii".

People said the same thing about the iPad.

Hi, buddy, I'm sorry to make you feel like we don't take women's voices into account. We came up with this name because RAG is already a consensus, as an acronym for retrieval augmented generation, RAG is used in many places, and we think it would actually be a standard for LLM oriented B-side scenarios, so that's why we adopted this name.

It only depends on the interface. There's a lot of projects which present the openai interface to whatever you want.

Not quite certain about your meaning. Could you be more specific? RAGFlow does not have its own LLM model or souce code. RAGFlow supports API calling from third-party large language model providers, as well as local deployment of these large models. RAGFlow has open-sourced these two parts of codes already.