Hacker News Reader

Comment by simonw | Hacker News Reader

On the one hand, this looks really useful.

On the other hand:

> There are various forms of attention / self-attention, Transformer (Vaswani et al., 2017) relies on the scaled dot-product attention: given a query matrix , a key matrix and a value matrix , the output is a weighted sum of the value vectors, where the weight assigned to each value slot is determined by the dot-product of the query with the corresponding key

There HAS to be a better way of communicating this stuff. I'm honestly not even sure where to start decoding and explaining that paragraph.

We really need someone with the explanatory skills of https://jvns.ca/ to start helping people understand this space.

View on HN

Complicated from whose perspective? I don't go around commenting on systems programming articles about how low-level memory management and currency algorithms are too complicated, or commenting on category theory articles that the terminology is too obtuse and monads are too hard.

I agree that there probably could be a better "on ramp" into this material than "take an undergraduate linear algebra course", but ultimately it is a mathematical model and you're going to have to deal with the math at some point if you want to actually understand what's going on. Linear algebra and calculus are entry-level table stakes for understanding how machine learning works, and there's really no way around that.

I am absolutely certain it is possible to explain this stuff without using jargon and mathematical notation that is impenetrable to the majority of professional software engineers.

I agree, but that doesn't mean that every article on the subject needs to be written for that audience in that language.

At some point, at some level, you really do need to just learn what a dot product is and what a matrix is. It's not weird notation or jargon, these are fundamental concepts.

Just like if you want to really learn how programs work you can't refuse any explanation that talks about "variables" or "functions" because that's jargon.

You can explain it, but it's going to be more at the level of "the network looks at the words" type explanation.

The idea of the transformer somehow being a trainable key-value store is kind of abstract and weird and has little to do with the mathematics of it. The math part of that is how the dot product encodes for similarity between vectors, but beyond that it really is a "if you get it you get it" kind of thing.

The Illustrated Transformer is pretty great. I was pretty hazy after reading the paper back in 2017 and this resource helped a lot.

https://jalammar.github.io/illustrated-transformer/

Thanks, that one is really useful.

Excellent. Thanks for sharing.

It took me a long time to wrap my head around the whole "key/query/value" explanation (and to be honest I regularly forget which vector is which). I find the "weighted sum of vectors" explanation much simpler/more intuitive; this blog post is IMO the best on the subject:

https://peterbloem.nl/blog/transformers

I found this article on “transformers from scratch”[0] to be a perfect (for me) middle ground between high level hand-wavy explanations and overly technical in-the-weeds academic or code treatments.

[0] https://e2eml.school/transformers.html

Vector, matrix, weighted sum, and dot product are good places to start. In fact, these concepts are so useful that they are good places to start pretty much no matter where you want to go. 3D graphics, statistics, physics, ... and neural networks.

Andrej Karpathy ie [0]?

[0] https://www.youtube.com/watch?v=kCc8FmEb1nY

Pretty sure the current state is simply one of confused exploration … hopefully soon to be synthesized and simplified.

Agree. Each new "advance" seems like just another trick/quirk (and adds jargon). Still waiting for the real advance.

Reply on news.ycombinator.com