> I don't know if you can "fix" hallucinations without changing the fundamental architecture.

Exactly. "hallucinations" are not some special case. They are the LLM working as designed.