One of the first things I learned in ML is that the model can only learn from the information in X. If X doesn't contain enough information to determine y, no amount of compute can fully recover it.

That's why I'm skeptical of grand claims about AI. Scaling can make models much better, but it can't create information that isn't there. An AI system can be extremely useful without becoming superhuman.