The idea that LLMs “only know how to predict the next word” is a common but imho wrong cop out. It has been shown that they are good at doing a lot of things that might not have been immediately obvious, regardless of how their internal process works. 50 years ago, somebody might say “programming is only a way to automate simple repetitive tasks” and that would be obviously wrong.

The real problem in cases like this and other applications, as you and many others have mentioned, is that LLMs are basically correlation machines. They can find very complex, immensely-multivariate correlations in large data sets, and reproduce these correlations very well. But they cannot reason (so far) beyond these correlations in other to find deeper, less obvious causal relationships. They’re simply not trained to do that, yet. But it’ll come…!