The library supports the model-graded factuality prompt used by OpenAI in their own evals. So, you can do automatic grading if you wish (using GPT 4 by default, or your preferred LLM).
Example here: https://promptfoo.dev/docs/guides/factuality-eval
Example here: https://promptfoo.dev/docs/guides/factuality-eval
"Robustness of Model-Graded Evaluations and Automated Interpretability" (2023) https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness... :
> The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability.
From https://news.ycombinator.com/item?id=37451534 : add'l benchmarks: TheoremQA, Legalbench