I think this is a misunderstanding of how these models work. The model does not understand anything at all. It's computing correlations. I could spend all day computing correlation without ever understanding what the correlations correspond to in the physical world. The correlations could still amount to a useful description of some physical phenomenon, but their interpretation requires much more than just the ability to compute correlations in some limited space.

If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.