Which is fine?
What's the point of getting people outside of the field reading a paper by essentially lying about the content? They come expecting A, and if they read it they understand it's actually about B. Loss of time for everyone except the author who wants to make a little buzz.
That's trivial and has already been done. GPT-3 didn't even get SOTA on SuperGlue.
These are the sort of misunderstandings that could have been avoided if the title was better.
In general, paper with "new variation of cloze pre-training task for this specific task" is a new section of the literature that is rapidly becoming sort of mundane and uninteresting because there are so many papers doing small variations of the same basic idea.
Of course neither GPT-3 nor the PET paper claim SOTA on SuperGLUE. They used a few-shot learning setup with 32 examples per task The normal SuperGLUE setup has hundreds or thousands of examples per task [1].
> In general, paper with "new variation of cloze pre-training task for this specific task" is a new section of the literature that is rapidly becoming sort of mundane and uninteresting because there are so many papers doing small variations of the same basic idea.
Could you please link to some of the work you are referring to?
[1] Table 1 in https://w4ngatang.github.io/static/papers/superglue.pdf