100 Pages of raw notes released with the language model OPT-175
twitter.com
twitter.com
They also swapped out the optimizer several times from what I can tell, switching between Adam, "Fake SGD", and "Vanilla SGD" multiple times.
Even without the huge amounts of hardware/driver issues they seemed to be having with the GPUs in their big training cluster(s), this puts into perspective how hard it is to train enormous models like this. Many of the failures don't have an immediately obvious cause. Plus, there aren't all that many places out there doing training at this scale so I imagine many of these things need to get figured out on their own.
After a quick look through, I really hope releasing raw notes like this becomes more of a trend!
I guess since their goal is to just be able to have a trained model it doesn't really matter. But it doesn't seem to be a easily reproducible process, and like i said a bit hacky in my opinion
Generally the way hyperparameters are adjusted is some mix of intuition/experience and random/grid searching. Plus most people don't have the resources/infra to do a large scale grid search on a model that might take a day or more to train. It's somewhat principled, but often a random search is just as good as fiddling numbers by hand and often you have to figure out why something worked post-hoc. You also accept that you might never have a good explanation - for all you know it’s dataset dependent - and trust that your results are good enough to convince peer review (and you can show that this other parameter set was worse, so you didn't use it). It's hacky in the sense that a lot of the work in getting to state of the art (moving the needle on a benchmark by less than 1%) involves playing with the numbers until you get the best results. For example here the engineers modify the learning rate between various runs. I don't think they really had any theoretical reason behind the step changes apart from "this will probably work better because we've seen that effect when training similar sized models".
Adjusting learning rate schedules is one of the simplest knobs to tweak. When you're working with huge models generally you want to use as big a batch size as you can get away with to reduce training time. A bit counter to the earlier thinking where LeCunn said something like "friends don't let friends use batch sizes > 32".
There may be some guided methods like exploring the parameter space in a Bayesian way (eg try to efficiently explore which knobs make the most difference).
Hideously ineficient and hacky to have someone manually tweaking things, but not terribly different from the state of the art for scientific research. As long as they state the objectives of their manual control and produce a log of what they did someone else could try to replicate it.
[0] https://github.com/facebookresearch/metaseq/tree/main/projec...
> CSP fat fingered and deleted our entire cluster when trying to replenish our buffer nodes.
Ouch.
I am oncall for a ~10k node system, and this log looks pretty similar to my workload... Yet Facebook only had 1% of the number of machines I look after for this! With far fewer machines, they should have far fewer failures!
I suspect they are doing a bad job of root causing failures to make sure they never happen again. For example, that Nvidia infoROM message should have ended up with all the logs and a couple of troublesome boards sent to Nvidia engineering to find out why the corruption happens, how to make it never happen again, how to scan to find out if it has happened, how to auto-undo the corruption, etc.
The same with the infiniband bandwidth issues - get that stuff sent to someone who can hook up a logic analyzer or look at traces to find out exactly why it's happening, and adjust the design of the hardware, firmware or software to make sure it can't happen again and that you have good visibility of any future similar issues beyond just 'its kinda slow, shrug.'.
> Goal: Get a 175B dense model up and running by any means necessary.
"by any means necessary" is engineering speak for "just keep solving problems, in the hackiest way possible, if necessary, and don't stop until the goal is achieved".
I didn't think ML model training ever needs on-call, especially for this kind of research-oriented ones. But apparently it's a thing. So is this what MLOps is about?