The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic.

I’m currently trying to implement chunking by topic using an LLM. It’s much slower, but I hope it will be a huge win in retrieval accuracy. First step is to extract topics from the document by asking the LLM to identify all topics, and then split by sentences and feed each sentence to the LLM to identify the topic. I’m hoping the result will be the original text, split by topics. From there, they can be further chunked if needed. Of course, it could be done by just one shot asking the LLM to summarize each topic in the document, but the more the LLM is relied on to write, the more distortion is introduced. Retaining the original text is the goal and the LLM should just be used for decision making.

Here is the crate I’m working out of. The chunking hasn’t been pushed, but you can see the decision making workflows. https://github.com/ShelbyJenkins/llm_client