I think you misunderstood my point, just as the other commentor misunderstood one level up.
Perhaps a different approach to explain the problem here:
"It ain't what they don't know, it's what they know for sure that just ain't so".
Perhaps a different approach to explain the problem here:
"It ain't what they don't know, it's what they know for sure that just ain't so".
c.f. Sally-Anne test: Sally thinks she knows where her toy is, we know that she doesn't, and indeed couldn't. The LLM (and humans in similar conditions) think they know what the distribution is, we know that they don't.
Only if you're interested in the specific failure modes that LLMs have.
That's all this story is.