LLMs work at all because of the high correlation between the statistically most likely response and the most reasonable answer.
On the same lines as why people argue if a tree falling in a wood where nobody can hear it makes sound because some people implicitly regard sound is the qualia while others regard it as the vibrations in the air.
An LLM should have no problem replying "I don't know" if that's the most statistically likely answer to a given question, and if it's not trained against such a response.
What it fundamentally can't do is introspect and determine it doesn't have enough information to answer the question. It always has an answer. (disclaimer: I don't know jack about the actual mechanics. It's possible something could be constructed which does have that ability and still be considered an "LLM". But the ones we have now can't do that.)
How often do the words "I don't know" get uttered in books, papers, articles, stack overflow, or any other resource of knowledge?
I have some representation in my mind; as someone who doesn't have aphantasia, this representation comes with a mental image. Tower? Tall, linear, and in my case a skyscraper by default. Eiffel Tower? Paying attention to the extra context, the first word transforms the second into the eponymous structure. Model Eiffel Tower? Now the context makes it a tchotchke, probably 10cm tall. Lego model Eiffel Tower? The 1-ish meter tall one on display in the Lego shop.
Is my "knowledge" the abstract representation that is in my case connected to a mental image? The attention process can reasonably be considered as developing a vector in a very high dimensional concept space, and the next token comes from what would best suit the current location in that high dimensional space. It's entirely possible that the concept of "ignorance" is linearly separable within that space (much as gender is, see the word2vec trick with "king" - "queen" ~= "man" - "woman"), and the corresponding "ignorance" vector can be associated with the sequence of words "I don't know". I think it would take actual research into the internal vector space to answer that, and while I'd like to do that research, I have some higher priorities right now.
Is "knowledge" any belief? Any true belief? Any justified true belief? https://en.wikipedia.org/wiki/Gettier_problem
I take the position that there is no such thing as knowledge, and instead the best we can have is belief.
But then, what is "belief", and can the information within an LLM said to meet whichever definition you give?
Answering "I don't know" because it a likely response to a particular string is completely different from being aware that one does not know the answer and saying so.
Both motivations lead to the same outcome, but they're unrelated processes. The response "I don't know" can represent either:
1. The most likely answer to a particular question, based on statistical data; or
2. An expression of an agent's internal state.
Figuring out that distinction is perhaps one of the most important questions ever raised.