The logbook is awesome: https://github.com/facebookresearch/metaseq/blob/main/projec...
This is the true secret sauce -- all the tricks on how to get these things to train properly that aren't really published.
This is the true secret sauce -- all the tricks on how to get these things to train properly that aren't really published.
100 Pages of raw notes released with the language model OPT-175 - https://news.ycombinator.com/item?id=31260665 - May 2022 (26 comments)
Some interesting takeaways:
- Meta aren't using any software for scientific logbooks, just prepending a document
- So many hardware/cluster issues.
- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)
- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project
I agree hot-swapping is underreported, but there are some good existing reports. In fact, OpenAI Five paper today is probably more valuable for its details on hot-swapping than its details on main model which used LSTM and not transformer.
> CSP fat fingered and deleted our entire cluster when trying to replenish our buffer nodes
Yikes
But why do they have so many problems to keep this cluster stable? Network failures? Bad GPU's? Bad drivers? Bad software?
Running fixmycloud and going after all those cryptic errors every day seems like a nightmare to me...
It makes sense to think of these things like Formula 1 cars, they are trying to eke out the absolute maximum performance, and reliability suffers because of that.
"Ordinary" cloud is more like a Toyota where you optimize for fuel economy and low maintenance.
These kinds of large transformers can be relatively reproduceable in results and benchmarks. However, making them converge to the exact same parameter set might not be a reasonable expectation.
Not much, but it also depends on what you mean by "reproducible".
Do you expect a similar internal representation (weights) or a similar behavior?