Hacker News Reader

Comment by ttul | Hacker News Reader

As with most tutorials on Transformers, this one leaves out some essential details:

- how are the input encodings generated?

- what is in those position vectors?

- how are the attention vectors learned?

The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a bunch of random numbers and then the numbers are gradually adjusted until the outputs converge closely enough on the desired loss.

I recommend watching Karpathy’s YouTube video where he codes up a Transformer from scratch. It’s the best way to understand these beasts.

View on HN

Thank you. This one?

https://youtu.be/kCc8FmEb1nY

I’m sure that’s the one being referred to. It’s great.

Is that an actual transformer, though? Like with encoder and decoder layers? That’s the part I never truly understood. Or is it “just” an example of a neural network? Thanks!

Not all transformers have separate encoders and decoders. GPTs, for instance, only have the equivalents of decoder layers of the original transformer paper, but they are still considered transformers. Karpathy’s video shows an actual GPT-style transformer.

I think a neural network can be considered a transformer if it contains a stack of attention blocks as its core mechanism.

The original transformer, from the "attention is all you need paper" had an encoder-decoder architecture because it was designed for language translation use where you are mapping one sequence to another, and are able to use both the preceding and following context of words when performing this mapping. The encoder utilizes the forward context.

In contrast to seq-2-seq use, for generative language models such as ChatGPT you only have access to preceding (not forward) context in order to decide what to generate next, so the encoder part of the architecture is not applicable and a decoder-only transformer is used.

Yes, that's increasingly the case, but there's no fundamental reason for nets to be trained end-to-end as a single entity.

Going back a few years it used to be quite common for people to use fixed word embeddings such as word2vec rather than learning them, and for image classification to take an ImageNet-pretrained general purpose model, then freeze the lower convolutional feature-detector layers and only train a new model "head" for more specialized use.

End-to-end learnt embeddings are going to be more optimal though, and in the context of these massive models the computational cost of training them is a drop in the bucket!

note that embedding and positional vectors are fixed and not trained with the transformer. The attention vectors are also computed in each step, but what is learned is the transformation of inputs to qvk vectors and the fully connected layers. There are some great people on youtube who explain the series of steps. I think this is the most comprehensive: https://www.youtube.com/watch?v=Nw_PJdmydZY

Thank you so much for setting me straight. CodeEmporium’s 10-part explainer is so great.

Reply on news.ycombinator.com