Some of their findings are really interesting, as well as their approach with the children stories, but I find it worrying that at no point the article mentions the problems inherent to the feeding AI models with the output of other AI models.

> The success of the TinyStories models also suggests a broader lesson. The standard approach to compiling training data sets involves vacuuming up text from across the internet and then filtering out the garbage. Synthetic text generated by large models could offer an alternative way to assemble high-quality data sets that wouldn’t have to be so large.

And they completely ignore the fact that those models are building their "synthetic stories" based on the knowledge from all that vacuuming text from across the internet, as if now we had already solved the problem of sourcing human data without the need for further vacuuming.