I never understood the thinking behind the separate key query value matrixes? What are they doing exactly?
It's a "fuzzy" hashmap; insead of hashmap.get("ball")=="threw" it assigns a probability to every pair of words.
Each hashmap captures some kind of relationship between words.
For example, every LLM has lots of heads whose relationship measures "is token1 the noun on which the verb token2 is acting"? So "I threw the ball" would have a high probability for ("ball", "threw").
But most of the hashmaps don't capture such easy-to-explain relationships. Some of them do. The rest probably capture relationships that we haven't figured out yet. This is the truly mysterious stuff.
But it's just hashmaps. Hashmaps all the way down.
Wq projects it to the space of queries. I.E What questions is this token asking?
Wk projects it to the space of keys. I.E What questions does this token answer.
Wk projects it to the space of values. I.E What are those answers?
Of course this explanation is prescribed onto the matrixes after the fact.
You can in fact do weird stuff like construct weights so attention calculates least squares, or sorts numbers, or other weird constructions like a transformer that calculates gradient descent steps. It seems to be very flexible in terms of what functions on data it can encode.
This sort of explanation, whilst being technically correct, is right up there with "A monad is a monoid in the category of endofunctors. What's your problem?".
Even if you have less background the jargon doesn't unpack as insanely as you might imagine. It's not quite the monad is a monoid level stuff.
A projection is just the linear algebra word for a function that takes vectors and produces vectors. And function in this case is just like a code function.
After that you need to believe that vectors (an array of numbers) can encode meaning abstractly (this is what "latent space" means). I mean this is sort of a given since what is a picture other than a list of numbers and obviously pictures can have meaning.
Finally you arrive at the statement that Wq is a projection (a function) from a token to the questions such a token could ask.
For example a token of "dog" could ask "what is the dogs name" as a query vector. Abstractly this is what Wq might do.
Maybe an earlier token "bella" turns into "this is a name" as a key. And "the name is bella" as a value.
The key and query in this case might line up, i.e directionally point the same way.
The real leap of faith is recognizing that this sort of abstract model of language can exist and be represented as just a vector of numbers. Which admittedly is a bit bizzare.