Well the most standard approach is to use cross-encoders (e.g. something like Cohere Rerank) to give similarity scores between the query and the chunk, and then use these scores to update the ranking.

Our approach is to use an LLM (gpt-3.5-turbo for example), and to ask it explicitly "Is this chunk <CHUNK> useful for answering this query <QUERY>". We've found, while certainly a bit more expensive, the larger model size and greater understanding of the world allows this approach to yield significantly better results that the SOTA cross-encoders. It also allows us to ask the model to explain why it's useful, which can be really helpful for the user when determining if they should look deeper into a document (as opposed to the standard keyword-based highlighting which often isn't very useful when determining if a document actually has useful information for your query).