LLM failure modes are caused by the lack of external context - they perform poorly on visual tasks because they have no sense of vision for example. Hallucinations are another aspect of this - as embodied agents, humans and animals have a strong bias for counterfactual reasoning because it is needed to survive in a complex information-rich environment (if you believe in something that is false, you tend to get eaten)

The real solution to these problems is to train transformers on a more human-like information context rather than pure text. Hallucinations should naturally decrease as LLMs become more "agentic"