> the web doesn't provide it in sufficient quantity and quality
Do you seriously think that "an AI-generated encyclopedia" would provide a better-quality training set? What would the "AI generator's" articles be derived from?
Do you seriously think that "an AI-generated encyclopedia" would provide a better-quality training set? What would the "AI generator's" articles be derived from?
This is the line of thinking behind the Phi lineup of models [0], as well as efforts to generate synthetic textbooks for training [1].
[0]: https://arxiv.org/abs/2309.05463
[1]: https://twitter.com/ocolegro/status/1712327588255809667
The second stage would be to generate research questions, then solve them with LLM+web search+code execution+other tools. The results would be compiled in reports. So it's a loop of problem generation, problem solving and validation. You can validate with highly trusted sources, or you can run code or simulations, ensemble multiple attempts, or even leave it to ranking by a preference model.