Not doing retrieval, but one thing I'd love to try would be active learning passes of querying. Here is a paper for large scale active learning

https://cdn.aaai.org/ojs/20591/20591-13-24604-1-2-20220628.p...

Steps would be:

- A first hybrid query with bm25+query vector

- rank some first results locally (ask the LLM for relevance, usefulness? ask the human?)

- query in active learning manner; providing positive and negative examples to orient the direction of better results. It'd silence the polluting components from the original query

- iterate

If anyone's hiring for remote roles, I'd love to give that technique a try