The Bayes-optimal text predictor would have to implicitly model the generative process underlying texts, that is, human minds. So it’s not crazy to think this approach could lead to an intelligent model.
Your example would maybe work if an intelligent agent had access to many compilers and reverse-engineered code bases. And if the algorithms were 1000x better than today. Otherwise it can guess until the cold death of the Universe without making progress.
Huh? A trained LLM is much 'wiser' than one that's just freshly randomly initialised.
You are right that they are still far from perfect, and we hope for future improvements in technology.