Some data is self annotating. You can count occurrences of features in context and then you know with increasing confidence that a feature occurs in a context. You can also build up meaning with more observation of events in context. Sounds circular, but notice no human is required in this process. Sure, in other cases or for other steps humans can be useful, but they aren't always needed for ground truth.

Hi, one of the authors here. Good question! For this benchmarking, we evaluated performance on popular open source text datasets across a few different NLP tasks (details in the report).

For each of these datasets, we specify task guidelines/prompts for the LLM and human annotators, and compare each of their performance against ground truth labels.