They use extensive model parallelism when training. Even TPUs (64GB) or Tesla V100 GPUs (32GB) don’t have enough memory to fit a model into a single child, so you’ll need activation checkpointing or model parallelism.
Realistically you'd be able to train up to 760M param models. For that you'd need 32gb+ VRAM GPUs which I think AWS might have. You can try iwth 16gb VRAM GPUs, but you would need to figure out FP16.
https://github.com/shawwn is doing some work in the GPT-2 space including using TPUs instead -- which has given him pretty good results.
32GB RAM is nowhere near what's necessary. It can barely fit the GPT-2 models on there with a batch size in the single digits. We'll need extensive model parallelism libraries (Zero2 from Microsoft) to run this at all.
I would hazard a guess that they will release versions that will be a smaller size (they have in the past). But in order to run this, you'd just have to use a cloud provider, first guesses say it'll be 500GB+ of just weights that ideally you want in memory.
Even the Teslas I use have 256GB memory and those are pretty cutting edge. 500GB in GPU memory is insane.
Could we bank on the Lottery Ticket Hypothesis, distillation, or other model compression algorithms to make these models smaller?
I would guess so, but compressing it by 1/3rd it's size (ie. distilgpt) would still be quite large. To be fair, I don't know if distillation scales like that.
Azure
This post may be tongue in cheek, but realistically that's right. I ran the Facebook chatbot that was posted maybe a month ago and didn't get good performance until I was using many Tesla v100 GPUs, which are $8k each. Thankfully modern CSPs like Azure allow renting rigs that can handle this pretty easily.
How much does that cost?
The largest AWS instances are $30-something/hr on demand, less with reserved time. So pretty expensive.
21.3 USD/h in vast.ai (8X Tesla V100, 118.8 TFLOPS)
More or less the cost of a well trained, experienced customer service representative?
Yeah but the cost will come down.
that's only 256GB, which isn't enough. I'm not sure it's even possible to nvlink 16 v100s. I'd love to try it out for $40/hr if it were possible though.