That's the black box.
The rest of the narrative implies there is a person inside. That's just what happens when you call something "AI": personification.
The ultimate irony of LLMs is that they are inference models, which means they can never "know" anything explicitly; but it also means that we can never "know" them explicitly.
Everything we have heard about inference models was itself inferred by humans! Do we truly have the same limitation, or can we take another approach?
I don't think that is the case. I don't think we are limited to modeling what the thing does through inference. I think we can construct an explicit understanding from how the thing is designed to work, because all of that exists as explicitly written algorithms.
We need to stop personifying the thing. We should probably also stop calling it a "Language Model", because that means we are studying the resulting model, and not the thing that constructs it.
I prefer to call them, "Text Inference Models". That's actually pretty easy to conceptualize: it finds patterns that are present (inference) in text (not limited to language patterns).
That gets us asking a more useful question: "What patterns are present in text?"
The answer is, "Whatever the human writer chose to write." In other words, the entropy of human writing.
That gives us a coherent source for what patterns an LLM might find. Some patterns are language grammar. Some are more interesting. Some are helpful. Some are limiting.
Most importantly, there is no categorization happening about the patterns themselves. Each pattern is on equal footing to the rest: each indistinguishable from each other. That means we can't ever choose the ones we like or avoid the ones that break stuff. Instead, we can only change what is present in the text itself.
Knowing that, it's easy to see where "limitations" come from. They are just the reality of natural language: ambiguity. That's exactly the problem inference models are intended to work around, but the problem could only be moved, not solved.