The other hard retrieval problems | Hacker News Reader

The other hard retrieval problems

softwaredoug.com

26 pointssoftwaredoug7 comments

What I got out of this is that you need hybrid search, and that involves reranking. I agree, and do not find the point controversial. Is there some thesis beyond this? I think the author needs to elaborate.

Since we're here, what libraries are y'all using for reranking in hybrid search these days? Anybody doing personalization or contextualization?

Quotidian viewpoint that everyone working in search already agrees with, for years, but nice to see it written down. Embedding-only search has never survived its first contact with an actual user.

Here's a bit of a quantification of the point Doug makes. Indeed, for a number of scenarios you get better results if you combine vector search and keyword search into a hybrid retrieval step, and do reranking on top of that.

https://techcommunity.microsoft.com/t5/ai-azure-ai-services-...

(disclaimer: I work in that team)

This is interesting. I recently built a search tool that needed to locate documents by keyword or by semantics, so I implemented a hybrid search straight away: BM25 + embeddings (from `gte-base`), with a cross-encoder for reranking.

I found that the lexical search was adding nothing; the embeddings alone produced almost identical results for keyword queries. (The re-ranker, however, made a big difference.)

Is this unusual?

It depends on the scenario. For example, for concept-seeking queries, vectors tend to do better (less likely to be an overlap in words between query and content), whereas for keyword searches (a product name, a serial number, project codenames, etc.) BM25 + keywords does much better. If your workload is all concept-seeking queries, it's reasonable that keywords don't add much.

If you look at the table in the section "3. Hybrid Retrieval brings out the best of Keyword and Vector Search" of that article, we shared there the significant variability of metrics as a function of query types.

If you ensemble a bunch of optimizations you get something that behaves like a neuron.

Neurons approximate the generator of the layers of 'blurs'. - The distribution.

Such generators have to have separate origin to be truly orthogonal. E.g. evolutionary algorithms for rodents, particle swarm for tool use. If you have orthogonality in your evolution generator, you are not modeling evolution.

To my intuition, when you are making orthogonal insertions into data with these constraints you are operating in a regime where computational complexity has exploded several times over what we are used to. Non-spoofed squirrels with screwdrivers would be extremely rare data so having an encoding scheme which allows for that would be a weird flex.

Thoughts still baking...

Yeah, you're not the first person noticing the Leopard Print Sofa Problem: https://redsails.org/suddenly-a-leopard-print-sofa-appears/

Reply on news.ycombinator.com