> Since AI is trained on "the internet" it sees and learns a lot of bad habits, poor coding examples, shortcuts, oversimplified solutions, and therefore, by default, produces code that can be improved.

This is a huge simplification. AI is also trained on tweets, spam and explicit content, but ChatGPT isn't constantly swearing at you to send your bitcoin to a Nigerian prince.

The mistake people make here is that just because something is represented in the weights, doesn't mean it's representative of its output. And the reason for that is largely the same reason for why if I show you lots of bad code you don't become a worse coder. If anything you'll see bad code and learn to avoid writing similar code yourself.

So long as quality is being reinforced and that process is refined and improved then "model collapse theory" and arguments about how AI will write bad code because it reads bad code will continue to prove incorrect.