Oh well, this is the death of democratic AI and an end of independent researchers :-( There goes any hope of a single Titan RTX producing meaningful commercial models.
That’s what 512 GPUs for 1.5 minutes costs on EC2, using spot P2.16xl with 16GPUs each. The price would be $12 if you had to use on-demand instances.
It does start to add up — if you wanted to do 2 runs per hour, 8 hours a day, 250 days per year (ie, a “full work year”), it would cost $48k. (And you’d have done 4000 such experiments.)
All of this is to say, I think you might be exaggerating this as the “death of independent researchers”.
A single card can still do meaningful inference on a model trained this way, and can do useful prototyping before you deploy your fanout for a few minutes and $20.
This also ignores all data transfer & storage costs.
There’s no network cost within a region, and you’re probably not downloading terabytes of data out of AWS — so you’re talking about a few dollars a month, as opposed to tens a day in GPU cost. (You’d also have 23TB of RAM in your cluster, if that’s enough for you to use as storage.)
Of course a single Titan RTX can't match the performance of a huge compute cluster, but I can rent a 512 TPU pod with 11.5petaFLOPS of compute and 4TB of HBM for $6.40 per minute. Members of the TensorFlow Research Cloud programme can access TPU pods free of charge.
Also you are talking about a model which is specifically designed for TPU (the dimensionality of the networks is especially fine-tuned).
And even tho, BERT_large still fit in the memory of a single GPU (for very small batch), there is an implementation on Pytorch. I don't understand, are people complaining that Deep Learning is actually (reasonably) scaling ? Isn't it a good news ?
MS already published another model based on BERT that is even better. It's unlikely memory x #GPUs would go down in foreseeable future; it's more like that everybody will start as large models as their infrastructure allows if they find something that improves target metrics.
We just submitted a paper, for instance, that is entirely CPU based, and required running 4 CPUs for a few days to reproduce (and even then, you can reproduce 90% of the paper within minutes on a single machine).
I was at an ML conference last year and I asked the panel: if I want to go forward in ML, should I rather do a PHD or work in the industry? a professor (!) answered that I should work in the industry as most research groups don't have the funding to be competitive.