Transformers from Scratch (2019) | Hacker News Reader

Transformers from Scratch (2019)

peterbloem.nl

63 pointssoftwaredoug9 comments

I'm not sure if it's just me, but this feels like a terrible article. It simply does not explain why "self-attention" is called self-attention in the first place, which would be very helpful for building intuition. Luckily, Wikipedia at least explains why "attention" techniques are labelled as "attention":

> In the context of neural networks, attention is a technique that mimics cognitive attention. The effect enhances the important parts of the input data and fades out the rest—the thought being that the network should devote more computing power to that small but important part of the data. Which part of the data is more important than others depends on the context and is learned through training data by gradient descent.

Now, it remains to figure out what the "self" if "self-attention" is for, but maybe someone else can fill that gap. ML isn't something I care about particularly, so don't want to spend too much time.

But that sounds exactly like the memory in long/short term memory recurrent neural networks (LSTM RNN)

The key difference is that RNNs store the already seen parts of the input to use it when processing future inputs, while self attention allows you to focus on all parts in parallel: information from the last input token can be used on the first token, which in RNNs is not possible (NB: that's one of many differences, but likely the one most relevant in this context)

It's called self attention because you only look at the input itself to determine which parts of the input to focus on. Transformers where first proposed for machine translation, and the neural net used in that publication consisted of two parts: an "encoder" part that takes an input language sentence and uses this self-attention to better understand it (and "encode" it into some neural representation), and a decoder which takes both the encoded source-language input as well as already translated parts of the target level sentence as input. These encoders use cross attention, ie they use the input sentence to determine on which part of the output to focus on next.

Dang, I've just been doing a bunch of reading about current transformers, and trying to understand how the output voltage is affected if the burden resistor is missing.

....but this is not that.

Guys, don't click the link. It's actually not about transformers, but in fact some machine learning BS (which, given the peak hype year of 2019 for ML, is not surprising).

Apropos real transformers, there is a new release candidate out for mtl 2.3: https://discourse.haskell.org/t/ann-release-candidate-for-mt...

No magnetic flux = no interest.

whats bad about ML transformers

Reply on news.ycombinator.com