It's talking about large language models and transformers so... yeah? (If you're talking about nanoGPT as a tutorial.)

nanoGPT is a way to embed low-dimensional embeddings into high-dimensional embeddings, not self-attention.