shabie

That's actually a pretty interesting point. Not just evals but other components like system prompt should also be tailored to match the expected outcome.

Thanks a lot for sharing, I have not heard of this before.

If I understood you correctly, yes I believe the notes should cover things that are not well-understood by LLMs more than stuff we know it typically gets right. So for us these are internal concepts and how people talk about them and less so about programming syntax.

Also, I've been thinking about adding a structure to the grading-notes so the variance in quality you get when asking people to leave notes becomes smaller. Yet, structure increases the burden... things like "what should the answer have", "what is definitely wrong to mention" etc.