For context: The latest large language models (LLM) skip doing some complex things that previous good models did, and focus on attention. See:
"Attention is all you need" https://arxiv.org/abs/1706.03762
"Attention is all you need" https://arxiv.org/abs/1706.03762
And for the Attention Is All You Need paper itself, Andrej Karpathy's "GPT From Scratch" talk is a really great walkthrough of how to make this work in practice, all condensed into two hours of live coding: https://www.youtube.com/watch?v=kCc8FmEb1nY