Quality control is often indeed by volume. When I was at Google / Facebook, search evaluations would typically have 5 raters each. One of the difficulties, though, is that search evaluation can be quite subjective. And honestly, most of the search engine raters we used were low quality and did a bad job (so the aggregate answer was often nonsense).