Potentially, yes! The scenario I am imagining is that context A as context for a query yields results 1, 2, and 3. Sometimes, finding neighbors of 1 (ie not necessarily in the top K w.r.t A) instead of going to results 4, 5, 6 might be better.