When we do math mistakes it is a form of hallucination just like for the LLM. In such instances you can ask a human to show their work and go step-by-step in which case they are less likely to "hallucinate."
Do the same with an LLM and likewise you'll get a better answer as expected. In your example though it is true you're unlikely to get a LLM to converge on the correct answer. Humans also struggle to reason well to some things beyond their understanding though.
LLMs can very well give their reasoning and steps and understanding for some things but not other things. Why isn't your response to this not that we just haven't taught LLMs the other things well enough yet and that we will probably get there soon enough? Would it be that surprising for GPT-5 to be able to show all of its mathematical reasoning for the question GPT-4 fails to answer now?
This is all besides the point. If AI is created that outperforms humans on everything and anything by several orders of magnitude then some response along the lines of "it isn't REALLY intelligent" just sounds like either some high-falutin' semantic jostling being done by an academic divorced from real-world implications or a human drinking some serious copium wanting to still feel special for being human.
I don't care how it works and that it is using processes completely different from human psychology. I care whether we can create AI in my lifetime that will outcompete me for any job I could ever hope to do. That and the totalitarian state implications of such power in the wrong hands.