> Once the top documents are retrieved, we ask a smaller LLM to decide which of the chunks are “useful for answering the query”
This sounds like normal re-ranking. How is it different?
This sounds like normal re-ranking. How is it different?
Our approach is to use an LLM (gpt-3.5-turbo for example), and to ask it explicitly "Is this chunk <CHUNK> useful for answering this query <QUERY>". We've found, while certainly a bit more expensive, the larger model size and greater understanding of the world allows this approach to yield significantly better results that the SOTA cross-encoders. It also allows us to ask the model to explain why it's useful, which can be really helpful for the user when determining if they should look deeper into a document (as opposed to the standard keyword-based highlighting which often isn't very useful when determining if a document actually has useful information for your query).
1. There's a difference between relevance and usefulness that cross-encoders cannot capture. Imagine a thread with a bunch of people complaining about an exception and each comment is another mention of the exception. Now imagine another thread with one mention of the exception at the top, and a bunch of people offering solutions. If you query for the exception, LLMs will find the second thread more useful, but cross encoders will find the first one more relevant.
2. LLMs/GenAI models don't output a single value. They can use the tokens they output to "reason" about the usefulness of a doc. Eg. Rerankers are like tiny LLMs that are only allowed to output "yes" or "no", but instead you can use an LLM to do chain-of-thought and finally decide at the end.