Historical common-crawl data [1] is available for download for free. Their data was the single most impactful source for GPT-3 [2]

[1] https://commoncrawl.org/the-data/get-started/

[2] https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilitie...