Why did we design LLMs to give us an answer at all costs vs. allowing them to say "I don't know?" I'd rather have a frank IDK than a hallucinated response because it's hardcoded to ALWAYS RETURN A RESPONSE.
Open your phone keyboard and just keep mashing the next predicted word until it stops predicting words. It will eventually collapse into a loop of predicting the same words in a nonsense sentence. Does it ever stop predicting "wrong" words?
That's literally how an LLM works. It predicts the next most likely word and that's it.