Hacker News Reader

Is that an actual transformer, though? Like with encoder and decoder layers? That’s the part I never truly understood. Or is it “just” an example of a neural network? Thanks!

View on HN

The original transformer, from the "attention is all you need paper" had an encoder-decoder architecture because it was designed for language translation use where you are mapping one sequence to another, and are able to use both the preceding and following context of words when performing this mapping. The encoder utilizes the forward context.

In contrast to seq-2-seq use, for generative language models such as ChatGPT you only have access to preceding (not forward) context in order to decide what to generate next, so the encoder part of the architecture is not applicable and a decoder-only transformer is used.

Not all transformers have separate encoders and decoders. GPTs, for instance, only have the equivalents of decoder layers of the original transformer paper, but they are still considered transformers. Karpathy’s video shows an actual GPT-style transformer.

I think a neural network can be considered a transformer if it contains a stack of attention blocks as its core mechanism.

Reply on news.ycombinator.com