chiccomagnus

IMHO, your article is missing an important point: 90% of implementations today flatten documents to plain text before chunking them. Why not consider the visual appearance that the human gave to the document? Using layout information combined with semantics, you can increase rag performances by +160% (tested via benchmarks), so why do most of us only use text?

Note: multimodal ≠ layout

How does credits works?

If you don't want to reinvent the wheel, we have built exactly that, goggle "Preprocess"

Have you compared this solution with tools like Preprocess, Reducto, etc.. ? I'm curious about the performance gain you can achieve with your approach

It seems it tries to always extract tables even if the content is just text. Is not working and at a similar price you can get high performing solutions like preprocess.co and similars.

I'm curious to know how you handle the chunking of complex documents

If for "documents" you mean PDF, Word, Powerpoints, Excel you should try preprocess.co

If that’s not what you meant, can you please clarify?

Do you use naive chunking? Have you tried something else?

By chance, have you tried preprocess.co for text extraction + chucking?

I see a huge missing point here: real world files are PDF, Word, PowerPoint and Excel, not only plain text.

There are more sophisticated chunking strategies, you will lose lots of context like that

Well-written article, missing key considerations:

- Titles matter, a lot: if you add the title of the section at the start of each chunk you will get 10x better embeddings and so more accurate results.

- The size doesn't matter: It depends on the combination of the layout and semantics of the content.

- Avoid garbage in / out: increased context windows don't mean you can put trash inside them. The more good you are at putting relevant information the more precise answers you get. Especially for enterprise-grade solutions, this is so important.

There are good emerging API solutions that implement semantic + layout-based chunking, which in my opinion is the best chunking strategy for PDF / Office files (the widest use case scenario for enterprises).

The real point that has been completely ignored is that data for real applications come from Office and PDF files, and making them plain text throws all the visual information.

Sorry, but it is really strange how haphazard it is. Am I doing something wrong?

It's missing more or less all the titles The order of the text is casual The system chunks at random points in the text

https://mlops.community/wp-content/uploads/2023/07/survey-re...

check out this https://preprocess.co

The datasets are composed by Documents like PDFs and/or Office files?

That's a good point, and documents too needs different chunking techniques. You don't want to split a word file the same way you split an excel...

You are both right about chunking, and i think is one of the main challenges. About more intelligent chunking approaches, i think you have to give a try to to preprocess.co It's able to preprocess and chunk PDFs, Office Files, and HTML content. It follows the original document layout considering the content semantics so you get optimal chunks

doesn't the basic chunking you are using for splitting documents have any impact on results?

It's an interesting topic i'm working on at the moment. Given the ones you shared, I figured out additional challenges. Especially for PDF and Office documents, it's really important to consider the original layout. You shouldn't split the text on a fixed word count based on the maximum length admitted by your embedding model. At preprocess.co we split documents considering sections, paragraphs, tables, images, lists, slides, and content semantics. The goal of a chunking method should be to generate text that increases the performance of a RAG not only to make it suitable for any text and file.