NanoGPT Speedrun

27 comments: https://news.ycombinator.com/item?id=49404380

>"This repository hosts the NanoGPT speedrun, in which we (collaboratively|competitively) search for the fastest algorithm to use 8 NVIDIA H100 GPUs to train a language model that attains 3.28 cross-entropy loss on the FineWeb validation set."

But, while very impressive (and it is!), that's not (IMHO!) what's so great about this repository... what's so great about it is the following list that starts out as follows:

>"This improvement in training speed has been brought about by the following techniques:

o Modernized architecture: Rotary embeddings, QK-Norm, and ReLU²

o The Muon optimizer [writeup] [repo]

o Use FP8 for head, and asymmetric rescale and softcap logits

[...] (etc., etc.)

See, that's like a who's who (or a what's what, as the case may be!) of

all the latest LLM training optimization techniques

In other words, it is a very information-dense list about optimization techniques for LLM training.

Think Mike Abrash, but 2026 and for LLM training...

In other words, a very high signal-to-noise ratio here!

Worth studying and re-visiting in the future for anyone into speeding up LLM training and/or people who just like algorithms in general... what a great list, and what a great tool!