Ask HN: Tips for Benchmarking LLM Performance?

So we went and got some local LLMs running. What we're missing now is a way to compare the performance of different foundation model, parameter and prompt combinations. I always see those benchmarks when a new LLM comes out, I wanted to ask if you could point me to some frameworks that can be used for benchmarking, maybe also to some theory? I'm not even fully sure how it works, I assume it takes a Q&A set, asks the questions, computes some distance metric between the example answer and the model output and then aggregates across questions (?)