TLDR: They say that transformers are pretrained word embeddings with one round of self attention that predict the next word sequence by beam search. They aren't any of those things.
Why start with word embeddings? The authors think that you first train word embeddings and then use them as the input for your Transformer. Nope.
> A word embedding can have hundreds of values, each representing a different aspect of a word’s meaning. Just as you might describe a house by its characteristics — type, location, bedrooms, bathrooms, storeys — the values in an embedding quantify a word’s linguistic features.
No. That's literally not what a word embedding is! It's completely the wrong intuition. It mixes up a semantic representation with the distributional hypothesis. These are two completely different ideas. Word embeddings are not describing words by their characteristics, exactly the opposite! They abandon that idea. To say that this is true but we just don't know what the characteristics being described are is just a total confusion about what is happening.
> Self-attention looks at each token in a body of text and decides which others are most important to understanding its meaning.
Why even introduce the word token here? When it's not explained at all. Just say word. And no, self attention does not decide which words are important for the meaning of this word, it's which other inputs are important to carry out some task. "meaning" is not a thing. Also, that's just attention. Where's the self part friends!? It sounds like they think self attention refers to a word itself paying attention. Things are getting dicey.
> With self-attention, the transformer computes all the words in a sentence at the same time. Capturing this context gives LLMs far more sophisticated capabilities to parse language.
Well, for one thing there's masked attention. For another, this isn't a property of self-attention. It's a property of how Transformers split apart attention and the positional feed forward layers. But I'm sure we'll talk about that (spoiler: we won't, things fall apart dramatically shortly; the authors don't understand anything).
> Transformers process an entire sequence at once — be that a sentence, paragraph or an entire article — analysing all its parts and not just individual words.
They just talked about an RNN that process words. And all of their parts. Also there are bidirectional RNNs. This comparison is nonsense. They should just not talk about RNNs if they can't say anything reasonable.
> This allows the software to capture context and patterns better, and to translate — or generate — text more accurately. This simultaneous processing also makes LLMs much faster to train, in turn improving their efficiency and ability to scale.
Ironically, they managed to identify the one part of LLMs that is slowest and most problematic as the part that is the fastest. A whole lot of papers focus on making attention faster because it's the computational bottleneck of Transformers; very funny to call that what makes them faster to train. Also, faster to train than what!? RNNs just don't scale up well (traditionally, let's not get into newer models). So it's not a matter of faster, it's a matter of, we couldn't train large models before at all.
Then there's a whole lot of "it does X, Y, and Z", no idea what in their story would make anyone think the models should ever do that.
> From this enormous corpus of words and images, the models learn how to recognise patterns and eventually predict the next best word.
Huh? How'd we sneak this one in? So the story they're telling is: we learn word embeddings, then we self attend to them which means looking at how other words related to them, and then that model learns to predict the next best word? Wow, it's like watching a blender process ideas into goo.
> After tokenising and encoding a prompt, we’re left with a block of data representing our input as the machine understands it, including meanings, positions and relationships between words.
tokenising!? encoding a prompt!? Where'd this come from. I love how we start the sentence with "tokenizing" and end it with "words", just a total confusion of ideas.
A block of data encoding the text as the machine understands it. Look at that beautiful block of data! A beautiful block of data that comes from.. word embeddings + positional encoding (huh? never talked about this one either) + self attention. Because, that's all there is to transformers.
Just some self attention, one round of it. On top of word embeddings.
That's the message. That's total and complete gibberish.
> At its simplest, the model’s aim is now to predict the next word in a sequence and do this repeatedly until the output is complete.
What is there to predict? In the story they've told I have pretrained word embeddings and self attention.
> To do this, the model gives a probability score to each token, which represents the likelihood of it being the next word in the sequence.
A "probability score"!? My inner statistician just took their own life. The next word in the sequence!? They're mixing up decoders with masked word prediction.
> With beam search, the model is able to consider multiple routes and find the best option.
Why!? Why talk about beam search? Who told them about beam search? Most models do something called top-p not beam search these days. But that's such a minor and worthless detail.
> This produces better results, ultimately leading to more coherent, human-like text.
I wish human-like text was coherent.
So there you have it. Transformers are pretrained word embeddings with one round of self attention that predict the next word sequence by beam search. cry