But the issue is, whatever the criterias are, they will become the main kpis opensource projects (at least the not-so-driven ones) will target for growth. Goodhart's law.
It's a bit like the SEO dance. Publishing the exact formula makes it much easier to game SEO, so instead search engines say stuff like "we're trying to gauge the overall quality of the site, we use metrics like [...] but not exclusively, focus on making good, responsive, accessible content above all else". Obviously it doesn't work perfectly and the more money there is, the more incentive to game the system, but it seems better than the alternative of publishing the exact ranking algorithm.
Of course, the big hazard here is "who watches the watchmen?" But public scrutiny plus the fact that others can jump in and try to take some of the same middleman role might help to keep that risk partly under control.