NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
qlabs.sh
qlabs.sh
Still, just for reference, here's the paper I remembered: https://arxiv.org/pdf/2507.15857
> Second-order optimizers and natural gradient methods
Do second order optimizers help improve data efficiency? I assumed they’d help you get to the same minimum faster (but this is way outside my wheelhouse).
https://arxiv.org/abs/2006.10732
The above provides a nuanced theoretical view. GD inductive bias is probably better unless your model is misspecified
If high-quality training data becomes the real bottleneck, then the interesting question is how much signal you can extract from the same dataset when compute is cheap.
0: https://entrpi.github.io/eemicrogpt/ (The writeup is from a few days ago, and I'm still running experiments before I do a big rewrite. Slowrun is good food for thought.)
The challenge is that such an approach almost certainly requires a model with RLHF post-training, but this needs to be done in the pre training phase. But with infinity compute, this isn't an issue - you simply do the post-training many times.
At the very least, track it. How will the project maintainers instrument this?
> Directions we think are wide open ... Curriculum learning
BabyLM and offshoot published a pretty convincing body of work on exactly that (which suggests it's not particularly relevant to LM training).
As I read your page, I really felt like the brevity-thoroughness tradeoff went the wrong way.
Ensembling makes multiple copies of the entire model, trains them independently on the same task, and then has every copy contribute to the output.
Reducing computation vs. increasing it; operating at per-layer granularity vs. whole model; specialization vs. redundancy.
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups."