The Benchmark Lottery
arxiv.org
arxiv.org
But what is the "original problem" and how do we measure progress towards solving it? Obviously there's not just one such problem - each community has a few of its own.
But in general, the reason that we waste so much time and effort on benchmarks in AI research (and that's AI in general, not just machine learning this time) is because nobody can really answer this fundamental question: how do we measure the progress of AI research?
And that in turn is because AI research is not guided by a scientific theory: an epistemic object that can explain current and past observations according to current and past knowledge, and make predictions of future observations. We do not have such a theory of artificial intelligence. Therefore, we do not know what we are doing, we do not know where we are going and we do not even know where we are.
This is the sad, sad state of AI research. If AI research has been reduced, time and again, to a spectacle, a race to the bottom of pointless benchmarks, that's because AI research has never stopped to take its bearings, figure out its goals (there are no commonly accepted goals of AI research) and establish itself as a science, with a theory - rather than a constantly shifting trip from demonstration to demonstration. 70 years of demonstrations!
I think the paper above manages to go on about benchmarks for 34 pages and still miss the real limitation of empirical-only evaluations in a field without a theoretical basis. That no matter what benchmarks you choose and how, without a theoretical basis, you'll never know what you're doing.
e.g. physics and chemistry took centuries of empirical fooling around to develop the highly predictive theories we have now.
I think one of the points of this article is that we need to think deeply about what our empirical foolings around are actually telling us.
I agree, but in that case most of the work done until now is inconsequential and a waste of everybody's time, and we should probably drop all of it and focus on setting down a solid theoretical foundation for future research. But there is lack of any incentives to do that and there are strong incentives to go on as we are now, from demonstration to demonstration of results and techniques that never get anywhere.
- _Algorithm A_ implemented by _Researcher A_ performs different to _Algorithm A_ implemented by _Researcher B_.
- _Algorithm A_ outperforms _Algorithm B_ in _Researcher A's_ study.
- _Algorithm B_ outperforms _Algorithm A_ in _Researcher B's_ study.
That's a simple case... and it can come down to many different factors which are often omitted in the publication. It can drive PhD students mad as they try to reproduce results and understand why theirs don't match!
[1] https://link.springer.com/article/10.1007/s42979-020-00265-1
>Sir, this is a Wendy’s.
I very much approve.
Congrats on beating the baseline! At this rate, surely you're only a few improvements away from winning the jackpot!
BER is a much more objective benchmark (even if alone it might miss power or some other factor). FWHM has it's own set of problems, because unless your an astronomer, the thing you're imaging probably isnt a point source. So you get into more subjective resolution phantoms, e.g. in medical imaging, or comparisons in how some part of Lena looks under different treatments.
But mostly,these benchmarks are all much simpler and objective than something like imagenet that has so much randomness in it, and no real underlying principle other than it being diverse enough to be general. I'm curious if there are other similarly random benchmarks used in other domains?