Evaluate is referring to measuring the accuracy of a model on a standard dataset for the purpose of comparing model performance. AKA benchmark.
Lots of ways to evaluate without humans. Most (nearly all) LLM benchmarks are fully automated, without any humans involved.