How is an A100 worth "only" 17 tera-flops?
There are multiple home gpus and even mobile chips rated at tera-flops. And the A100 really is a gigantic chip.
I realize all about bandwidth, and full vs half precision, and inflated marketing vs actual benchmark.
But can a 10-node-iphone-cluster really match an A100 for pure flops?!
Only if the data is local, small, easily synchronizable, doesn't require fp32 precision and a dozen other caveats.
Can a 10 iphone cluster match an A100 in calculating digits of pi? Yes. Can it do backpropagation in large machine learning models? The gradients won't even fit.
It's optimized for TF32, INT8, bfloat16, and other similar data types that the tensor cores like.
Double precision is important for a lot of simulation codes because of numerical issues. For Deep learning applications it's basically unused, single precision or half precision formats dominate training. The same numerical issues don't show up, so this is a lot more efficient.
Additionally a lot of the 'AI' flops are achieved through matrix cores, fitting Deep learning the workload well. This brings an A100 from only 17 to ~200 tflops.
> But can a 10-node-iphone-cluster really match an A100 for pure flops?!
The iphone number is probably fp32 if not lower precision. However the combined SoCs of 10 iphones have tripple the transistor count and a larger chip area among them. So the comparison is not totally off.
https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...