You don't get these types of generational improvements in the CPU world these days. In some ways though,it’s better that the gains go to the GPU because, after years of talk, we actually do seem to be in the middle of some really awesome ML developments.
Not fast enough (not at the price point) but getting there: of course since it's an entirely parallel problem, the real metric is cost-per-model since in reality you'd just buy time from the cloud.
I figure we're going to start seeing some big changes once cost-per-model puts it within reach of the hobbyist. If I today could buy a custom model for say, $1000 - that's accessible to the hobbyist experimentalist.
that has been disclosed :/
I think you have to compare it to how much ETH you could have mined as the opportunity cost of training a model vs other uses of a GPU
I was more contextualizing this, where a potential 75% increase in power (on a smaller node) to deliver 50% more performance is less impressive:
> I think it's even more impressive that the smallest increase vs. A100 is still 50% (for ResNet)
If I were deploying 8-GPU nodes today, I'm not sure whether I would want to pick between 4-GPU nodes or doubling my node TDP. The DGX A100 and H100 both have 8 cards, and 640GB of VRAM, but peak power increased from 6.2kW -> 10.2kW. If you're training giant models and constrained on VRAM, you'll potentially need just as many nodes as before just to fit your models into memory. Your training will be faster but it's hard to avoid the power increase.
The power increase is much more reasonable if you stick with PCIe SKUs (300W -> 350W, literally half the power draw of the SXM SKU), but you pay for that with 20% less compute and 33% less memory bandwidth.
That's misleading, as the GPU power utilization during ResNet is very likely not 100% for either the A100 or H100.
Remember the lower-TDP PCIe H100 has 20% slower compute and 33% slower memory than the SXM model, suggesting the increased power delivery from the PCIe (350W) to SXM (700W) model is a major factor in the performance even for H100 vs H100.
I don't think it's misleading to say that power is extremely likely to be a factor in the demonstrated performance increase from A100 to H100 for non-transformer workloads until proven otherwise. I don't think anyone here has a DGX H100 in hand yet to test this.
Edit: wait, also I looked closer at the numbers - in MLPerf 2.1, NVIDIA only submitted results for 1x H100. There's no 1x A100 result submitted for resnet50, just an 8x A100 number, which NVIDIA seems to have divided by 8 to get a "per accelerator" number to compare to their 1x H100. That doesn't feel like a clean comparison, as you can have performance loss when scaling. I'd rather see 1v1 or 8v8.
It might be easier to grasp if you use 1x. E.g. "it's 1x as fast" means it's exactly the same. You're applying the multiplier directly to get the new speed. 0.5x as fast means it's half as fast.
"It's 1x faster" means you add the result of the multiplication to the initial value. So it's twice as fast. 0.5x faster still means it's faster, you add 50% of the initial speed. This way you cannot really express that something got slower, except by using negative values, which might be rather confusing.
I think this might also work in most other languages originating in Europe.
To my ears, "3 times faster" has the same meaning as "3 times as fast".
Some people want to use language that is logical and consistent. For them, "3.5x faster" means the same as "350% faster" and "4.5x as fast". Others believe in convenience and redundancy. They think that "4.5x faster" means the same "4.5x as fast", because the numbers are the same. Many of them don't like expressions such as "X% faster" for X >= 100, because the numbers tend to be misleading.
I used to be in the former camp when I was young. Today I'm middle-aged and lazy. When I see "3.5x" in the text, I assume that it means "3.5x". I don't want to read the text carefully to determine that you actually meant "4.5x" when you wrote "3.5x". I interpret the "faster" part as redundancy. It tells me that we are talking about speed and that the thing we are talking about is faster than the baseline.
You could say we are mid-journey…
- Significantly-improved diffusion models (DALL-E 2, Midjourney, Stable Diffusion, etc)
- Diffusion models for video (see https://video-diffusion.github.io/, this paper is from April but I expect to see a lot more published research in this area soon)
- OpenAI Minecraft w/VPT (first model with non-zero success rate at mining diamonds in <20min)
- AlphaCode (from February, reasonably high success rate on solving competitive programming problems)
- Improved realism and scale for NeRFs (see https://dellaert.github.io/NeRF22/ for some cool examples from this year’s CVPR)
- Better sample efficiency for RL models (see https://arxiv.org/abs/2208.07860 for a recent real-world example)
I'd add GPT3 & Github Copilot, which my team and I use professionally. It's far from perfect, but it's a great GSD tool especially for stuff like Regex, bash scripts, and weird APIs.
reasonably high was 50% on 10 attempts, meaning success rate on first attempt can be as low as 5%, out of which who knows how many were leaked to training data.