Paste in some text and switch to the token IDs view. Note how common words (like "the ") have low integer token IDs, while things like emojis are split into several numbers.
An LLM is a function that takes an array of integers and returns a new array of integers. Seeing the tokens like this helped me reinforce that mental model.
To refine this a bit more, a LLM is a function that takes an array of integers (or really, a batch of arrays of integers), and returns a probability distribution for each possible integer, with the array shifted left by one place to enable prediction.
The integers are really indeces into the embedding space.
So you'd want to think more that the model maintains a giant matrix (one row = one token; one column is an embedding feature).
The array of indices gets the relevant embeddings to shove through the rest of the model's forward pass.
I don't understand that, because wouldn't the probabilities later in the sentence be impacted by the tokens chosen earlier in the sentence?
Yes, what happens later in the sentence depends on the particular choice you made earlier in the sentence.
More recently, there are models like RWKV that can run in both parallel (GPT-like) mode for training and serial (RNN-like) mode for inference.
But transformers always output a probability distribution at each position in the context.
The API docs talk about letting you specify your own stop token (like "<!-->") but I don't think "token" is meant in the same sense here.
A score like this can be useful for active learning though, where you find areas of low confidence in your dataset and get more data to train on.
What I still can't wrap my head around is that tokens often don't align with word structures.
"Antidepressants" I'd imagine tokenizes as "anti" "depress" "ant". But nope. And "antipsychotic" tokenizes differently from it too!
I assumed the output is a token i.e. a single integer and that's rarely even a full word?
The prompt is initially loaded at the start of the list and the model is run and produces high activation on a single output. That token output is then fed to the end of the input circular list and also added to the "this is what the model returned."
This process of running the model, getting the token output and sending one copy to the input list and one copy to the return string is repeated until the number of tokens generated hits a numeric limit or a token that represents the stop token is encountered.
Tokens are symbols. You're thinking of them like embedding vectors. Tokens represent the step before a meaning is assigned to the text: it turns some unit of text into what's essentially an identifier.
Which is to say, two homonyms would have the same token id, even though they have different meanings. Tokens have no notion of context.
1. Greatly reduces memory usage. Instead of memorizing every inflection of the word "walk", it memorizes the root (walk) and the modifiers (ing, ed, er, ...). These modifiers can be reused for other words.
2. Allows for word compositions that weren't in the training set. This is great for uncommon or new expressions like "googlification" or "unalive".
If you put:
test walk walker walking walked
[test][ walk][ walk][er][ walking][ walked]
I added "test" to that because walk at the start doesn't include the leading space and [walk] and [ walk] are different tokens.
For even more fun, [walker] is a distinct token if it doesn't include the leading space.
test walker floorwalker foowalker
[test][ walk][er][ floor][walker][ fo][ow][alker]
(Late edit)
walker floorwalker
[walker][ floor][walker]
Generation is ultimately deterministic (seeded prng) so backtracking wouldn't make sense.
I found Stephen Wolframs explanation helpful. He has a YouTube video version which I enjoyed too. This blog post was on HN last month, but I never get good search results on hn
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
Next token prediction produces the most head exploding emergent effects.
You can put people in an fMRI and ask them to think "car".
You can ask someone to think of objects and detect when they think "car".
What happened there pairing a bunch of tensors to meanings and matching them.
We can do something similar with embeddings.
To be clear I don't intend to give the impression that these LLMs are doing something miraculous. Just that we are increasingly peeling back the veil of how brains think.
I don't know about other people, but when I think “car” really hard, I can feel the muscles in my throat adjust slightly to match the sound of the word “car”. Perhaps that sort of thing is what the MRI machines is picking up, rather than being able to pick up some kind of "internal representation" of car.
It'll also light up in the parts of my brain to do with reading, writing, hearing the word in the languages I speak.
What does car mean to me if it doesn't connect to all the concepts that relate to cars?
Instead of coming up with more and more heuristics to chop a sequence of bytes up in "words" in a vocabulary, we could simply set a limit on the size of the vocabulary (number of tokens), put all bytes in there (so we can at least handle any input byte by byte), and pack the remaining space with the most common multi-byte byte sequences. Then you end up with tokens like here.
There's no "better way" to do it because the tokens are all meaningless to ChatGPT, it only cares about how efficiently they can be parsed and processed.
The competing desires are to model all language with the biggest tokens possible, and the fewest tokens possible. The lines aren't meaningless, text is split into the largest possible chunks using a set of the most common tokens.
Common words, like "the", "fast", "unity", "flying" are all tokens, but it's not because they're words, it's because they're common letter clusters, undistinguished from "fl", "ing", "un", "ple"
"gadflying" is tokenized into [g, ad, flying], even though it's only loosely semantically related to "flying", it's just the most efficient way to tokenize it.
'this is a day that this sentence with clarify that day. Is this not a good day?'
[5661, 318, 257, 1110, 326, 428, 6827, 351, 18282, 326, 1110, 13, 1148, 428, 407, 257, 922, 1110, 30]
Note 'day' is solidly 1110 here. Now start a sentence with day.
"day began with laughter"
[12393, 2540, 351, 20263, 13]
So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well.
"Day after day after this day"
[12393, 706, 1110, 706, 428, 1110]
"Day day Home home home"
[12393, 1110, 5995, 1363, 1363]
"day day day home home home"
[820, 1110, 1110, 1363, 1363, 1363]
[corrected/edited: so case-sensitive and position sensitive as well.]
btw doesn't the output array contain the prompt as well (because of the transformer architecture? not entirey sure ~iirc)
You're missing that it groups in spaces. The position isn't relevant, but "day" is a different token than " day".
p.s. "ifthiswasagermanword if this was a german word."
It's not even spaces. That second sequence ' german' is chopped up as 'ag' 'erman'.