Hacker News Reader

Comment by shaklee3 | Hacker News Reader

The fp16 results versus the 1080ti are somewhat surprising. The author specifically pointed out that they are using tensor cores were possible. I would have expected fp16 would be more than 100% faster than the 1080ti if they were using tensor cores. Can anyone explain that?

View on HN

Not everything in the compute pipeline is going to be converted to fp16 operations. Anytime you are doing accumulation or exponentials, you would have to have it in fp32.

There was a good talk from NVIDIA at last years GTC: http://on-demand.gputechconf.com/gtc/2018/presentation/s8923...

Here is another relevant blog post: https://devblogs.nvidia.com/mixed-precision-training-deep-ne...

EDIT: Also not everything in the training loop is a matrix multiplication where tensor cores are useful.

Right, but the tensor cores should be about 10x faster on the compute side, and about 2x the memory bandwidth. GEMM is usually constrained by compute, which is why the tensor cores exist.

These benchmarks are for training, so the expectation is that they are running them in fp16 all the way through. Also, tensor cores can accumulate in fp32 registers with a slight hit to performance.

Is there a benchmark you've seen that matches the claimed 10x increase in performance for the Tensor Cores? NVIDIA hype train can sometimes make it difficult to find hard numbers.

I've done my own benchmarks where I've hit over 100 TFLOPS on the V100, and that's about 85% of the peak theoretical throughout of them. Granted the matrix size needs to be large enough, but it's definitely doable. Anandtech also showed similar results in their V100 review. I haven't yet seen a comparable SGEMM done on the 2080Ti, so I don't know how it'll compare. I have some coming in at the end of the month though, so I should know soon.

Titan V in my testing only approached 100+TFlops in purely synthetic large matmuls.

With carefully tuned Transformer (matmul-heavy!) I could only make twice as fast as 1080ti (at 4 times the price).

The only undisputable benefit was using double batch size.

Turing tensor cores are not as fast as Volta ones. You won't see the 10x number repeated on any turing marketing material.

What's your source for that? They should be the same speed, but more capable. You absolutely see the 8x in their marketing: https://devblogs.nvidia.com/nvidia-turing-architecture-in-de...

I think a good answer to this is here talking about the tensor cores on the Nvidia Titan V: https://github.com/tensorflow/tensorflow/issues/15897#issuec...

If you can write a better optimized network, go ahead. But like in SSE2 vs AVX2 vs AVX512 benchmarks, FP performance on paper doesn't always translate into better real world FP performance. Now if Nvidia had switched to HBM2 like Google TPU2, it might be different.

It's still about 60% faster than the 1080 ti, which is huge in my opinion.

Reply on news.ycombinator.com