Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it.

I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means.

Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes).

Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget