Your Queries Can Reorder the Retrieval Leaderboard

Short version: I used a few retrieval benchmark datasets, and experimented with a 21 embedding model leaderboard. I held the benchmark corpus and relevance labels fixed, and transformed the queries: paraphrase, terse, verbose. I kept the “information need” the same under each query transformation, but the leaderboard order changed more than I expected.

I reproduced published benchmark scores for benchmark dataset + embedding model pairs, and then used that same setup with transformed queries as the input. I observed the reordering under each rewrite style to be consistent across 4 independently-generated rewrites (gemini-2.5-pro, claude-opus-4.8, gpt-5.1, llama-3.3-70b) rather than showing up in just one.

Code and query score files are here so you can check the tables without a full re-run: https://github.com/georgian-io/query_form_experiment

Let me know your thoughts on this … or in general let me know about any experience with putting an embedding model into a production retrieval system and being surprised by poor performance.