the issue is that the large language model doesn’t understand what it’s saying, cannot reason or come up with novel ideas, yet it can convince a human that it _is_ doing so.

so then if the world begins to prefer AI-generated content, any question you ask the internet will only show you AI-generated answers, which can only offer you answers based on its training set, and over time a system with no new inputs just ends up being static generated from static, albeit static which is convincing enough for a human to accept it.