The Ultra-Scale Playbook: Training LLMs on GPU Clusters

Kudos to Hugging face for going the extra mile on open sourcing knowledge. They could have "just" created a hub for model downloads, but the fact that they keep publishing top-tier tutorials, materials, experiments and code that benefits the whole community is such a blessing. IMO they're one of the most important players advancing the open source community effort. Keep it up!

Ok this is way too long to read in one sitting, but it looks incredible? I've been looking for resources like this for """real""" model training at scale. The authors have worked on BLOOM (the first >100B parameter model), starcoder and fineweb so they actually know this space.

I wish it were not a single giant webpage, but I guess I can deal with that.

Hugginface is mostly AWS, so these experiments might be done on an AWS-provided cluster? I wonder how many of the results are reproducible on "open market" clusters listed on e.g. https://gpulist.ai/