There is a well known problem with LLMs that if you feed it its own output, it gradually gets worse and worse.

We haven't really understood what the limitations of LLMs are. And I do not know. But as a person who uses Fable and Sol regularly to design my own new programming language, they suck at the task of defining new systems coherently. So far no AI I have tried is good at defining new coherent systems well. I suspect it's because of the recursive problem of using it on its own output.

So, a human is still needed to define and write the sort of first genesis of the system, and then AI can take it once the problem has been defined. But defining the problem and using it on itself is what AI is really bad at (for now).