Evaluating RAG for large scale codebases

LLM as a judge isn't telling you if something is right or wrong, it's telling you if a given generation is normal or an aberration, according solely to the data the model was trained on. This is one reason LLMs prefer their answers to answers from other LLMs.

"LLMs as a judge" is more about addressing the failure mode of auto-regressive (one-token at a time) generation letting an LLM lead itself astray due to its previous choices, rather than telling you any general truth.

Finally I'd note that in every maths challenge I ever completed as a student, you were strongly advised to go back and check your own work at the end if you had time left over, and for me this usually led to me catching things I'd missed the first time.

No, it's like a teacher's assistant using a answer sheet provided by the professor to grade exams. A common practice at the university I went to and accurate enough in most cases.

The problem of matching an answer to a given answer is a lot simpler than generating an answer. Especially for an LLM which has language transformation as one of its core competences.

...pretty well according to student.

It's student writing judge evaluation questions for himself as judge, judging judge (himself) and evaluating himself later using own judgement (as judge).

1 student all the way down.

If it thinks eating concrete makes you stronger, it's going to think that and give green light end to end.

One of the most effective things to do in coding (and many other things) is to always do a self-review: Before you turn over a changeset for review, review it as if you were reviewing some one else's code.

It sounds superfluous, but it works very well as it saves time and frustration by allowing you to quickly fix the stuff that wouldn't pass review anyway.

The LLM as a judge-concept works in a similar way. Instead of "give a good answer", the task and perspective is "does this make sense?", which is very different.

It's mentioned in the article that they used their own repos.