A cartel of influential datasets are dominating machine learning research

> The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.

Edit: By "they", I was actually referring to the linked article. Strangely, even the paper the article is about does not cite "The Benchmark Lottery" at all.

Here's the paper that this article is recapping - "Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research", https://openreview.net/forum?id=zNQBIBKJRkd

Abstract: "Benchmark datasets play a central role in the organization of machine learning research. They coordinate researchers around shared research problems and serve as a measure of progress towards shared goals. Despite the foundational role of benchmarking practices in this field, relatively little attention has been paid to the dynamics of benchmark dataset use and reuse, within or across machine learning subcommunities. In this paper, we dig into these dynamics. We study how dataset usage patterns differ across machine learning subcommunities and across time from 2015-2020. We find increasing concentration on fewer and fewer datasets within task communities, significant adoption of datasets from other tasks, and concentration across the field on datasets that have been introduced by researchers situated within a small number of elite institutions. Our results have implications for scientific evaluation, AI ethics, and equity/access within the field."

The reviews seem quite positive, if short. On a skim it looks very solid, offering an empirical look at the dynamics of benchmark usage that IMO seems unprecedented, so I'm not surprised it got positive reviews and accepted.

The word cartel implies collusion. There is no collusion.

Using the word "cartel" in this case is not only wrong, but also... insulting to all the hardworking individuals who have toiled away in obscurity to compile, clean-up, label, package, and publish influential datasets.

Despite the terrible title, the OP is worth a read. It summarizes a recent research paper, also worth a read, about the disproportionate popularity in ML research of a relatively small number of datasets sponsored by and produced at elite institutions (e.g., ImageNet for visual recognition):

https://openreview.net/forum?id=zNQBIBKJRkd

Surely we can find ways to address the winner-take-all dynamics of benchmark popularity without unfairly accusing anyone of running a cartel.

Are these big "dominant" institutions charging for this data? No, the spend a lot of resources putting them together and give them away free.

Are they preventing others from giving away data? No, but it costs a lot and they bear that cost.

Are they forcing smaller institutions to use their data? No. Its just a free resource they offer.

Do they get the grants themselves because they have some kind of proprietary access? No, the whole point is that the benchmark is open and everyone has access.

So they collect this data, vet it, propose their use for benchmarks and give it away free. What is the complaint? The problem is not even posed as "well, this data is overfitted in papers or we are solving narrower problems". [Edit: they sort of do, but leaving this here so the comment below makes sense as a follow up.]

No the complaint is that these handful of institutions are giving away free data so too many people use it? Can we have more "problems" of this nature?

The alternative is either a small dataset that people heavily overfit (eg the MUC6 corpus that was heavily used for coreference at some point where people cared more about getting high numbers than useful results) or things like the Universal Dependencies corpus which are provided by a large consortium of smaller institutions

In fact, as far as I can tell, we are not overfitting in this sense. When I have seen papers examine whether progress on, let’s say, imagenet, actually generalizes to other categorization datasets the answer is yes.

What we have been seeing is that the slope of this graph is flattening out a bit. Whereas in the past a 1% improvement on imagenet would have meant a 1% improvement on a similarly collected dataset, nowadays it will be more like .5% (not exact numbers just using numbers to illustrate what I mean by diminishing returns.)

If an institution or a lab can show that progress on their dataset -better- predicts the progress on a bunch of other closely related tasks, then as researchers become convinced of that, they will switch over. Right now there isn’t a great alternative because it’s not easy to create such a dataset. Scale is critical.

Imagenet really was on the right track as far as collecting images of nearly every semantic concept in the English language. So whatever replaces it will have to be similarly thorough and broad.

In my opinion the biggest weakness of currently existing datasets is that they are typically labeled once per image with no review step. So I think the answer here isn’t

“Let’s get researchers to use smaller datasets from smaller institutions”

It would be more like

“We have to figure out a way to get a bigger, cleaner version of existing datasets and then prove that progress on those datasets is more meaningful”

The realistic way this plays out is that some institution in the “cartel” releases a better dataset and then lots of small labs try it out and show that progress on that dataset better predicts progress in general.

As it turned out, differences between datasets proved significantly larger (by a big margin) than differences between algorithms. And the most popular datasets in fact included biases and eccentricities that were bound to cause such problems.

https://bastibe.github.io/Dissertation-Website/ (figures 12.9-12.11, if you're interested)

The problem here is, though, most publishers don't accept papers if the new proposal isn't backed by benchmarks by the well-known datasets. Even though it can be a competitive approach for a specific field, they reject the paper anyway if it's not performed well on the datasets.

Granted, datasets have grown larger and larger over time. This concentration is actually a healthy sign of a community maturing and converging towards common benchmarks.

These datasets are open to all for research and have fueled considerable progress both in academia and industry.

The authors would be well-advised to look at what is happening in the community. Excerpts from the program of NeurIPS happening literally this week:

Panels: - The Consequences of Massive Scaling in Machine Learning - The Role of Benchmarks in the Scientific Progress of Machine Learning - How Copyright Shapes Your Datasets and What To Do About It - How Should a Machine Learning Researcher Think About AI Ethics?

All run by top-notch people coming from both academia and industry, and from a variety of places in the world.

I am not saying that everything is perfect, but this article paints a much darker picture than needed.

For example, if there are 50 datasets of historical weather data how can I determine which one is garbage?

Now we have exploded that corpus to include everything anyone says online, and we are worried that the lack of curation means we cannot be sure what the models will come back with.

Its rather like sending our kids out to find an education themselves, and finding three come back with nothing, two spent years learning from the cesspit of extremism, two are drug addicts hitting "more" and one stumbled into a library.

Just a thought but I dont think journalism is really about writing newspaper articles. It really is about curating the whole wide world and coming back with "this is what you need to know". Journalism is the curation AI needs...

'A fact beats theory every time...And I think economists have focused on theory because it's easier than collecting data.'

When I read this article this is exactly what I thought of -- modeling/researchers always focus on the low hanging fruit which is sitting in their comfy chair in their ~200 year old university developing hyper-complicated models rather than going out and collecting data that would answer their questions.

I'll have to read the paper to see what exactly it says on this but my knowledge of NLP benchmark datasets is exactly the opposite: the majority are simply no use for measuring the capabilities they're supposed to be measuring. For example, natural language _understanding_ datasets are typically created as multiple-choice questionnaires, so that, a) there is already a baseline accuracy that a system can "hit" just by chance (but which is almost never noted, or compared against, in papers) and b) a system that's good at classification can beat the benchmarks black and blue without doing any "understanding". And this is, indeed, what's been going on for quite a while now with large language models that take all the trophies and are still dumb as bricks.

To make matters worse, NLP also doesn't have any good metrcis of performance. Stuff like the BLEU scores are just laughably inadequate. NLP is all bad metrics over bad benchmarks. And NLP results are much harder to just "eyball" than machine vision results (and the models are much harder to interpret than machine vision models where you can at least visualise the activations and see... something). I think NLP is much, much worse than machine vision.

https://paperswithcode.com/dataset/debatesum

https://huggingface.co/datasets/Hellisotherpeople/DebateSum

https://scholar.google.com/citations?user=uHozRV4AAAAJ&hl=en

Minor typo in this article...

ARTICLE: Among their findings – based on core data from the Facebook-led community project Papers With Code (PWC) – the authors contend that ‘widely-used datasets are introduced by only a handful of elite institutions’, and that this ‘consolidation’ has increased to 80% in recent years.

...but right after, they quote the paper and clearly it is 50% not 80%. See the quote from the paper:

PAPER: ‘[We] find that there is increasing inequality in dataset usage globally, and that more than 50% of all dataset usages in our sample of 43,140 corresponded to datasets introduced by twelve elite, primarily Western, institutions.’

...and the article is leaving out this relevant quote from the paper:

PAPER: Moreover, this concentration on elite institutions as measured through Gini has increased to over 0.80 in recent years (Figure 3 right red). This trend is also observed in Gini concentration on datasets in PWC more generally (Figure 3 right black).

...and in general the article is right that inequality is increasing over time, but Gini is a specific metric to measure inequality, and 0.80 is not the same as 80% inequality.

Edit: After asking a friend seems most of their research is with Chinese official research institutes and Sensetime. It makes sense now.

[1] Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, https://arxiv.org/pdf/2103.14749.pdf

Once solved, these no longer are research areas. Getting X score on a Y validation set means you get your s/w drivers license.

For a computer science analogy: It is a paper with the finding that most successful computer languages are created at prestigious institutes. An obvious -not a bad - finding. Not like you could give the motivation, skills, expertise, resources, and time to a small new institute, and expect these to come up with a new language which the community will adopt.

Yes, if you write and publish a good data set, and it gets adopted by the community, then you gain lots of citations. This reward is known, and therefore some researchers expend the effort of gathering and curating all this data.

It is not a "vehicle for inequality in science". Benchmarks in ML are a way to create an equal playing field for all, and allows one to compare results. Picking a non-standard new benchmark to evaluate your algorithm is bad practice. And benchmarks are the true meritocracy. Beat the benchmark, and you too can publish. No matter the PR or extra resources from big labs. It is test evaluation that counts, and this makes it fair. Other fields may have authorities writing papers without even an evaluation. That's not a good position for a field to be in.

> The prima facie scientific validity granted by SOTA benchmarking is generically confounded with the social credibility researchers obtain by showing they can compete on a widely recognized dataset

Here, authors pretend social credibility of researchers has any sway. There is no social credibility for a Master's student in Bangladesh, but when they show they can compete, then they can join and publish. Wonderful!

Where the authors use the long history of train-test splits, to pose the cons have outweighed the benefits, they should reason more and provide more data to actually show this and get the field to get along. Ironically, people take more note of this very paper, due to the institution affiliation of the authors. I do too. If they had a benchmark, I would have first looked at that.

> Given the observed high concentration of research on a small number of benchmark datasets, we believe diversifying forms of evaluation is especially important to avoid overfitting to existing datasets and misrepresenting progress in the field.

I believe these authors find diversity important. But for overfitting, these should look at actual (meta-) studies and data. This seems conflicting. For instance:

> A Meta-Analysis of Overfitting in Machine Learning (2019)

> We conduct the first large meta-analysis of overfitting due to test set reuse in the machine learning community. Our analysis is based on over one hundred machine learning competitions hosted on the Kaggle platform over the course of several years. In each competition, numerous practitioners repeatedly evaluated their progress against a holdout set that forms the basis of a public ranking available throughout the competition. Performance on a separate test set used only once determined the final ranking. By systematically comparing the public ranking with the final ranking, we assess how much participants adapted to the holdout set over the course of a competition. Our study shows, somewhat surprisingly, little evidence of substantial overfitting. These findings speak to the robustness of the holdout method across different data domains, loss functions, model classes, and human analysts.

Oh boy. PWC is not even close to a representative sample of what datasets are being used in papers. It's also often out of date.