Hacker News Reader

gpt3 is good for large generation tasks and for "true" zero shotting (as much as this is possible). people know this. this is a weird title

View on HN

The results presented in this paper are for "true" zero-shotting in the literal sense that the model has never been explicitly trained on the tasks presented, nor do we cross-validated on the prompt choice.

don't you pretrain on very silar tasks explicitely

We discuss this a bit in Section D.2 (HOW UNSEEN ARE THE HELD-OUT TASKS?). From our perspective,

a) The tasks we test on are very different, particularly tasks like BIG-Bench that we didn't even have access to until several days ago (and none of us read).

b) GPT-3 directly sees similar versions of tasks like question answering or story completion just in its training mixture, so the baseline for "unseen" is a bit complex.

Minor correction: I (Stella Biderman) am a contributor to BigBench, have read many of its tasks, and have had access to it for months. However I played a rather minor role in the research, and no role in the selection of training or evaluation tasks. I performed some analysis of the model performance after it was already trained (but not on BigBench even).

Reply on news.ycombinator.com