Hacker News Reader

Comment by swyx | Hacker News Reader

this was the deepest dive we ever did into Transformers Math 101 (https://news.ycombinator.com/item?id=35631546). we mostly cover AI Engineer/inference time things on the pod, but training knowledge is hard won and extremely valuable, and I love it when experts are wiling to distill the rules of thumb for what they have learned.

questions welcome!

View on HN

Sorry for the criticism here, but your linked paper / article does nothing to explain the math behind transformers. You should re-name it something like 'Scaling Transformer Mathematics' or 'The Math Behind Scaling GPUs / Transformers' or whatever more descriptive name there is to describe what you're outlining in your article. 'Transformer Math 101' should at least explain 1) input embeddings 2) (keys, queries, values) and the idea behind the matrix multiplication of these linear / numeric sets of values 3) the Softmax computation in the original paper as well as the relevant matrix transformations which take place 4) dot products ( and how they relate to similarity scoring) 5) feed-forward neural networks and the gradient descent / backpropagation and how they work. There are many many concepts you didn't even touch upon. This is not 'Transformer Math 101' by any means.

fwiw i'm not the author of that doc, we just interviewed him, and the hn submission i linked to was also renamed presumably for similar concerns. we do have an "Algorithms 101" episode (in the theme of our Datasets 101 and Benchmarks 101 episode) where we have at least some of your topics lined up

related online discussion when this article came out:

from Eleuther: https://twitter.com/AiEleuther/status/1648782486736969728?s=...

from AI anon: https://twitter.com/nearcyan/status/1662937711156625408?s=20

Stella Biderman (coauthor and now lead of the Eleuther Foundation, Stella if you're reading this pls come on our pod sometime!!!!) https://twitter.com/BlancheMinerva/status/164878587828883456...

I'm quite busy right now, but maybe in late September or October?

Yes! I have a 5 pages doc of prep notes I made, happy to share :)

Please do! Thanks

Reply on news.ycombinator.com