I am always reminded of Severance, the TV show, when I think of LLMs. Data refinement is what is happening massively right now. Basically, take data harvested on the Internet, which may be of poor quality, and use a teacher-student approach to reformulate it in higher quality. Then use that synthetic data to train new models. It's a bit like the biblical multiplication of the bread loaves. How much farther can it get LLMs? I don't know.

During the next turn, though, a lot of the same synthetic goop is going to turn up on the Internet. That's probably why OpenAI is spending the big bucks to secure access to original content producers. Now, here is to hope "original content producers" are not going to depend on LLMs to generate that original content.

I think training data would be plentiful if autonomous drones roamed the face of the earth, constantly recording audio and video. There is nothing as good as reality to learn what real is. The representation of reality on the Internet has become a matter of opinion for a lot of people.