What is, then, the qualitative difference between a GOFAI automated reasoning system [1], like Coq, and an LLM? Why does Coq do reasoning but GPT-4 doesn't? Does it only count as reasoning if hand-coded by a human? Or did you have another definition of "reasoning" in mind?
Statistical or probabilistic reasoning is still reasoning, and indeed essentially mandatory to make sense of the real world, and is what humans do. Indeed the failure of painstakingly hand-trained symbolic reasoning engines to model reality (as opposed to just simple toy worlds) was the entire reason for the rise of neural networks.
The entire point of a neural network is to efficiently compress large amounts of statistical data into a model with as much predictive power as possible. Somewhere within GPT-4's many layers there has to exist something like Bayes networks and other ultra-efficient representations of the vast network of correlations present in the training corpus – otherwise there's no way it could do what it does.