mingtianzhang

Thanks, that’s a good point. Yeah, it makes retrieval look like function calling or tool selection, which I guess makes the idea more generic and better suited to current AI systems like MCP.

For the ToC generation quality, you can try our API: https://docs.pageindex.ai/ or the open-sourced version: https://github.com/VectifyAI/PageIndex. I didn’t realize other people were working on similar ideas or had similar packages. It would be great if you could share the links to the AI agent you mentioned. Thanks!

Yeah, that’s a good point. I’m starting to like this term. It would also be great to read the paper you mentioned—could you share it with me?

But keyword search is at least a white-box method, it gives you what you want. But semantic similarity is an approximate search and unexplainable.

Yeah I should call it 'Vibe RAG' lol.

Exactly — thanks for the insightful comments! The goal is to generate an “LLM-friendly table of contents” for retrieval, rather than relying on vector-based semantic search. We think it’s closer to how humans approach information retrieval. The table of contents also naturally produces semantically coherent sections instead of arbitrary fixed-size chunks.

- Corpus of documents: Yes, this approach can generalize. For multiple documents, you can first filter by metadata or document-level summaries, and then build indexes per document. The key is that the metadata (or doc-level summaries) helps distinguish and route queries across documents. We have some examples here: https://docs.pageindex.ai/doc-search

- Question-agnostic indexing: The indexer does not know the question in advance. It builds the tree index once, and that structure can then be stored in a standard SQL database and reused at query time. In practice, we store the tree structure in JSON, and also keep (node_id, node_text) in a separate table. When we get the node_id from the LLM, we look up the corresponding node_text to form the context. There is no need for Vector DBs.

- Handling large tables of contents: If the TOC gets too large, you can traverse the tree hierarchically — starting from the top level and drilling down only into relevant branches. That’s why we use a tree structure rather than just a flat list of sections. This is what makes it different from traditional RAG with flat chunking. In spirit, it’s closer to a search-over-tree approach, somewhat like how AlphaGo handled large search spaces.

Really appreciate the thoughtful questions again! We’re actually preparing some upcoming notebooks that will address them in more detail— stay tuned!

Now all the headings can be rendered properly in your markdown visualization if using this OCR tool.

Hi, I think our approach can benefit from these OCR tools to generate a better tree for search. We will update more on this point in our GitHub repo, thanks for raising this question!

Hi Imanari! That’s essentially one of the key challenges we’re aiming to address with our PageIndex package.

We’ve designed two LLM functions:

a. LLM Function 1: init_content -> initial_structure

b. LLM Function 2: (previous_structure, current_content) -> current_structure

The idea is to split a long document into several page groups (each within the context window size). You first apply Function 1 to the first group to get the initial structure, then use Function 2 in a for-loop over the remaining page groups to recursively build out the rest of the structure.

This approach is commonly used in representation learning for time-series data. We'll be releasing a technical report on it soon as well.

Mingtian

Hi Bsenftner, thanks for your interest.

The motivation behind building PageIndex is to build a reasoning-based RAG. When we previously designed a RAG system for financial documents, we encountered two main challenges:

1. Traditional embedding-based RAG often returns redundant information because all the financial terms are semantically similar.

2. We want to incorporate expert experience into the RAG process—specifically, experts often have a preferred order of where to look first.

To address these, we developed PageIndex, which transforms long documents into a structured “table of contents.” This allows the LLM to selectively retrieve relevant nodes based on reasoning. With this approach, we can do few-shot learning by providing examples of expert preferences directly in the prompt, enabling the LLM to choose nodes more like a domain expert would.

-----

In your case, it sounds like you're looking for a system that can automatically build and continuously update a knowledge base as new data arrives. You might benefit from something like:

1. Using expert knowledge to define a template knowledge graph—e.g., specifying entity types, link types, or a rough graph structure.

2. Building an agent that updates the knowledge graph when new documents are received. The agent’s tasks could include:

a. Identifying new information relevant to existing nodes or links.

b. Determining whether this new information changes the current knowledge graph.

c. Updating the graph accordingly.

Since your use case involves logical reasoning (not just semantic similarity), PageIndex and reasoning-based RAG could play a helpful role here. In other words, while a traditional graph-based RAG might still be used at inference (question answering) time, PageIndex and reasoning-based RAG can assist during the knowledge graph update phase by identifying related information in the new documents that are related to the graph. Additionally, the tree structure produced by PageIndex can be used as an initialization for building your knowledge graph.

Hope this is helpful! Mingtian

Hi, thanks for the comment! I agree with you that the sections are very useful. In the end of the blog, we did mention that "In practice, an improved chunking strategy should also consider the boundaries of sentences or paragraphs. We'll discuss how to integrate this consideration into this method in our upcoming blog post. " Stay tuned!

1. The limited context window of LLM poses challenges when summarizing large documents. A common strategy (provided by LangChain or LlamaIndex) is to: (i) divide the document into chunks, (ii) summarize each chunk using LLM, and (iii) combine them into one global summary using LLM.

2. For document division, a bad chunk size can result in the last chunk being much smaller than the previous ones. As a result, the global summary might overemphasize the last chunk and thus bias away from the core semantics of the document. See https://tinyurl.com/vectifyai-summarization for an example.

3. In our blog, we introduce a straightforward method to automatically decide the optimal chunk size for different documents. We find this new strategy significantly reduces bias in the global summary. The implementation can be found at https://github.com/VectifyAI/LargeDocumentSummarization.

Stay tuned at Vectify AI (https://vectify.ai) for more innovative techniques to turbocharge your LLM pipelines!