„It's not magic just yet, models aren't purely performing context-based retrieval. They're leveraging their pre-training knowledge extensively. Always be careful when running retrieval experiments on items that might be well represented in the training data (read: the entire public Internet). Results for never before seen data might be much less impressive.“
This is a really important message regarding the power of LLMs.