Garbage in Garbage out is real and shouldn't be underestimated. Its about as useful as trying to understand quantum physics from the perspective of a flat-earther
For example, AlphaZero uses synthetic data (playing against other versions of itself) but it does gain information because the data is generated against the rules of the game.
Similarly, word2vec and other vector representations use synthetic training data, but that data is derived from observations of text (no, it's not using raw text, each training instance is generated from the text, with varying levels of simplicity of course).
Tesseract is trained using synthetic data generated from font files and simulated "rendering errors."
Synthetic data should not be underestimated.
There was also a paper on persistent computer programs after running random data iteratively over millions of cycles. This doesn't even have a 'goal'. which by itself is very interesting because of how similar it feels to how organisms develop by the act of persisting after a few billion years.
That is to say I largely agree with the premise that you can train and improve alot by absense of data. But if your original dataset is garbage, then training over and over might get limited by a hidden nash equilibrium that the algoritm can't escape due to lack of new information.
Obligatory Anton petrov: https://www.youtube.com/watch?v=L_IWVZPmc-E Paper: https://arxiv.org/pdf/2406.19108
Wouldn't it make more sense to say trying to understand general relativity from the perspective of a flat-earther? At the quantum level, the world is flat--I mean, ontologically. "Dimension" is not used to describe space and time in the same way as in Einsteinian physics which retains a very Kantian model of the cosmos. But then you could certainly re-interpret Kant to make it cohere with quantum physics, I'm thinking of people like Deleuze and Latour here, which is why I alluded to a "flat" ontology.
But in truth, it sort of negates your point, since when it comes to gathering things from the totality of the productive process (which is what an LLM does), indexing them and creating a surrealist wet-dream of output, there is not going to be a "right" answer, in just the same way that a quantum physicist and a cosmologist are not going to be able to agree on the correct definition of time. Truth is at the level of feeling, not sense. No scientific instrument can detect the feeling that one thing caused another, even though we are absolutely sure of its continuity, and it is such continuity that stands as the basis of all scientific investigation. In the end, empirical laws, just like moral laws, simply "feel right," and that's it. The LLM tells us things that feel right. It is a good-feeling machine, yet it is precisely when we notice something wrong, when it seems totally off, in those sublime gaps, that it becomes the most enigmatic and, in fact, the most useful. The real beauty of the AI is in those gaps, in its imagination, because it is actually our own, projected outward and reflected back like a society-wide mirror. There is something wonderful in the dreams of a machine.