BM42: New Baseline for Hybrid Search

I was doing this trick with word embeds in 2017 - cosine similarity all to all, then take average similarity for each word to the others to rank them, then use these ranks for a weighted sum to embed the document.

But the fundamental problem of embedding is that they refer to the raw text, not its meaning. For example "What is the most frequent letter in this question?" won't embed like "t". Calculated inferences don't get embedded, only the surface level. You can't search for "funny scenes" in a book, unless the jokes are explicit.

This means we need to use LLM to "flesh out" the kind of things we are likely to search later. Raw text must be studied before indexing. And if you need, reindex to update the analysis as you extend the scope of queries.

Going even deeper, any new piece of information added to a corpus might shed new light on everything else, think of a plot twist for example. The analysis stage needs to iterate and aggregate data until it stabilizes.