The first thing that popped into my head is what do you do with the test results? Specifically, how do they feed back into model improvement in a way that avoids overfitting? Do you think having some kind of classical "holdout" question set is enough? Especially with RAG, I'd wonder with the levers that are available (prompt, chunking strategy, ...) if you define a bunch of test questions do you end up overfitting to them, or to the current data set. How can findings be extrapolated to new situations?