The paper opens by saying "Large language models (with more than 1 billion parameters) perform well on a range of natural language processing (NLP) tasks in zero- and few-shot settings, without requiring task-specific supervision," and cite Radford et al. (2019), Brown et al. (2020), and Patwary et al., (2021) for this sentence. But the first paper doesn't claim zero-shot performance and the other two sources are about models orders of magnitude larger! I don't see any evidence in this paper to support the idea that there are people going around claiming that 1B+ parameter models have impressive zero-shot performance.
They make a similar error that reinforces this one in their conclusion as well. They say "At first sight, these models show impressive zero-shot performance suggesting that they capture commonsense knowledge," completely ignoring the fact that they did not in fact ever show that. They also never explain how large their "SOTA" model is, which seems quite important. They additionally never compare to the models that are actually claiming zero-shot performance. Their model performs similarly to GPT-3 13B on HellaSwag, 1.3B on Winogrande, and 6.7B on PiQA. There's clearly some large confounding factors they aren't controlling for here.
The fact that ML researchers, and especially NLP researchers, use extremely low quality baselines is not news. People publish papers pointing this out all the time. The same is true of the fact that many NLP datasets are garbage. "PiQA and HellaSwag are bad evaluation metrics" is potentially a worthwhile inclusion to the literature (I haven't checked if these particular datasets have been critiqued) but is something personally known to me and something that no NLP researcher should find surprising. If people are surprised by this, I think that those people really need to spend more time reading the literature and evaluating datasets. Your default assumption should be that a benchmark eval is loosely correlated with what it nominally measures. And all of these things really have no bearing on zero shot generalization.
If the primary value of this paper is pointing out that the datasets are bad, that can manifest as misleading zero-shot scores but it also manifests as misleading few-shot scores and misleading fine-tuned scores. I would expect a finetuned and few-shot version of Fig 6 to look pretty much the same, but we are not shown such plots.
Why? I can't be sure, but the authors take themselves to be specifically criticizing zero-shot claims and if those plots look as I expect it would significantly undermine their claims. And even if they don't, these models are not being evaluated in a regime in which people are actually claiming significant zero-shot performance. The paper's entire framing is predicated on the false claim that people are claiming that 1B+ parameter models are zero-shot learners.