An equivalent number of V100's (GPT-3's original GPU) would've taken about 36 days [0].
0 – https://www.reddit.com/r/GPT3/comments/p1xf10/comment/h8h3sl...
https://twitter.com/abhi_venigalla/status/167381386318645248...
(300BT / 1.2BT) * 11 min * (1 hr / 60 min) = 45.8 hr
An eight GPU DGX-1 server cost ~149k$ back then (googled news postings). A current gen DGX H100 is 520k$ with 5 years of support. Of course it holds 5x the memory, plus GPUs and interconnect are much faster. But when comparing costs, take price hikes into account.
For writing code you don't care about feeding world history to your model. So a smaller model might be better at a specialized task
Sure, having a big multi-modal-model is great, but by having specialized models you can spread tasks better
Just to compare the GPUs, TF32 Tensor processing went from ~125 TFlops to ~990. It then looks like they also dropped the precision to FP8, which gives you another 4x win.
What's interesting is to look at how we're progressing in performance over time. In some sense, a bit slow?
A V100 costs $10k at release; an H100 seems to be $40k.
So we've only managed to halve the cost of a flop in 5 years. That seems.. much slower than what Moore's Law would have suggested.
On a (minor) side note, it seems that $1.00 from 2018 is worth ~$1.20 in 2023. I wish more cost comparisons included inflation, because the past few years have had a lot of it.
You need to be an order of magnitude higher in power usage before it really starts mattering. e.g. a Tesla3 consumes around 15 KW (20x the H100) while driving.
That said it looks like flops/watt dropped by only 3.4x, which is also sub Moore's Law (3 years to halve power consumption)
You couldn't replicate the scale of compute this allows no matter how many V100's you had.
A huge amount of cost here is embodied in networking and memory.
If one were to design a chip that cared only for flop/$ without caring for all of the interconnect and memory, then the 4090 is a much fairer comparison, and even then that card isn't designed for a flop/$ optimisation.
Moore's law says nothing about price or performance.
H100 has about 4x as many transistors as V100, which is pretty close to what Moore's law would predict.
Moore's law is about doubling of transistor count for the same price. At least that's always been my understanding.
EDIT: I decided to look it up. Heres the original 1975 statement from him that led to the law:
"The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years."
So yup, it's about transistor density vs price.
The V100 had up to 32gb ram, the H100 is 188gb. While there’s obviously transistors in ram, the counts being compared are for the GPU itself. I’d argue that a big chunk of the price difference in the V100 vs H100 is RAM.
But that's the same Moore's Law problem when using the 'price' definition.
6x RAM 5 years later = 4x the cost? That's even slower than the FLOPS gains.
Roughly the same tech advances apply to making all three kinds of chips so I think it applies (somewhat) equally to CPU, GPUs, RAM and even SSDs etc.
Bearing in mind that ram lags a few generations behind CPU and GPU (I think DDR5 is 12nm vs latest GPUs around 4nm). And also that Moore's law is not a physical law, even when it was in full swing it was only ever meant as a rough guideline for what to expect over a couple of years period.
If you bought the $2/hr H100 instance they offer, that would cost ~$330k. Pricey, but not too bad.
My bigger hope is that with cheaper compute we can see more architecture search and designs. A lot of different architectures are relatively unexplored due to computational constraints and are typically performed by smaller labs so they don't scale and it is kinda hard to compare models when we're just looking at performance benchmarks and not considering other factors. We definitely don't want big labs to railroad our research directions. Feels weird that a huge amount of NLP is based on using pretrained models and tuning them. Vision is going this way too. You basically can't get published without being SOTA so you basically have to modify an existing model or have a multi-million dollar lab and train from scratch. Really weird to expect academia to compete with big labs and really weird to not let academia take "bigger risks" and explore less popular areas. It is vital to our research path that we don't force everything onto a single track.
3,584 GPUs at $30,000 is $104,550,000 USD only in GPUs.
That’s why these clusters are so valuable, even with Nvidias margins they are still cheaper than using less compute for longer.
that means they might pass a few batches through a GPT-3 sized random init network and time it