This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words you intend to weight them against.
There is a phenomenon called Broca's Aphasia which is, essentially, the inability to connect words into sentences. This mostly prevents the patient from communicating via language. But patients with this condition can reveal quite a bit about the structure of the language they can no longer speak.
One example discussed in The Language Instinct is someone who works at (and was injured at) a mill. He is unable to produce utterances that are more than one word long, though he seems to do well at understanding what people say to him. One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!".
This is the opposite of what you describe, a single token that is longer than one word in the base language instead of being shorter. But it appears to be the same kind of thing.
By the way, if you study a highly inflectional language such as Latin or Russian, you will lose the assumption that interpretive tokens should be whole words. You'd still expect them to align closely with sentence structure, though.
I'm sorry, but I'm lost on how that's a single-word utterance.
There's an article here where the structure of an image generation network is explored a bit:
https://openai.com/research/sparse-transformer
They have a visualization of what the different layers are paying attention to.
There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful:
Tokens are so closely tied to modern LLMs that’s it’s basically impossible to not talk about them. They’re getting a lot of attention because they are the primitive. They’re the thing of most interest for improving performance.
If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step.
People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority.
That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".
Just trying to provide an example.