Except, that the LLMs are only working when the instructions they are "understanding" are in their training set.

Try something that was not there and you see only garbage as result.

So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning.

But they do have a LOT of examples in their training set, so they are clearly useful. But for proof of reasoning, I want to see them reason something new.

But since they are a black box, we don't know, what is already in there. So it would be hard to proof with the advanced proprietary models. And the open source models don't show that advanced potential reasoning yet, it seems. At least I am not aware of any mindblown examples from there.