GPT-3 was pretrained on five datasets (Common Crawl, WebText2, Books1, Books2, and Wikipedia; see table 2.2), and then used on previously unseen tasks (Q&A, translation, cloze, etc.) without finetuning, i.e., weights were not updated after the original (autoregressive) pretraining. This promises a possible future in which general-purpose models are pretrained once, and deployed to production for multiple tasks.