Code is logic. LLMs are expressed with logical syntax. I recommend meditating on this fact for some time and trying to understand its implications instead of buying into the hype about LLMs being world models.
LLMs aren't world models. They're linguistic models. However, with the right training database, it turns out that a good parsimonious linguistic model will have a (multiple, probably) world model as a subset, as it is so much better at compressing/predicting the textual stream.
Logic is part of the real world. Logic is not the real world. It does not scale to all of reality. This is obvious. anything without clear definitions and unambiguous axioms and it falls apart. You don't have to take my word for it.