Hacker News Reader

Comment by savant_penguin | Hacker News Reader

1 petaflop on a chip?? What is the catch?

Tensor petaflops are useful in only very few circumstances. One of which is the highly lucrative deep learning community though.

The main tensor op is a matmul intrinsic which is useful for way more than just deep learning.

Edit; many of these speeds are low precision which is less useful outside of deep learning, but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.

> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.

The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.

You are indeed correct, I was (kinda) fooled by the marketing and I think that TF32 is deceptively named. I think the tensor cores are being used in this architecture for FP64 and 60 TFlops is still pretty decent.

I'm on Turing architecture so I've never used TF32. I've only used FP32 and FP16 but FP32 isn't supported by these tensor cores.

Well the addition is done in FP32, and it's a 32-bit storage format in memory, so calling it a 16-bit format isn't right either. It's really a hybrid format where everything is 32-bit except multiplication.

Given that it's 32-bit in memory (so all your data structures are 32-bit) and also that in my experience using it is very transparent (I haven't run into any numerical issues compared to full FP32), I think calling it a 32-bit format is a reasonable compromise.

> Well the addition is done in FP32

Addition is done in 10-bit mantissa. So maybe TF19 might be the better name, since its a 19-bit format (slightly more than 16-bit BFloats).

Really, its a BFloat with a 10-bit mantissa instead of a 7-bit mantissa. 10-bit mantissa matches FP16, while the 8-bit exponent matches FP32.

So TF19 probably would have been the best name, but NVidia like marketing so they call it TF32 instead.

It's a 32-bit format in memory and the additions are done with 32-bits.

I admit that I don't have the hardware to test your claims. But pretty much all the whitepapers I can find on TF32 explicitly state the 10-bit mantissa, suggesting that this is at best, a 19-bit format. 1-bit sign + 8-bit exponent + 10-bit mantissa.

Yes, the system will read/write the 32-bit value to RAM. But if there's only 10-bits of mantissa in the circuits, you're only going to get 10-bits of precision (best case). The 10-bit mantissa makes sense because these systems have FP16 circuits (1 + 5-bit exponent + 10-bit mantissa) and BFloat16 circuits (1 sign + 8-bit exponent + 7-bit mantissa). So the 8-bit exponent circuit + 10-bit mantissa circuit exists physically on those NVidia cores.

-------

But the 'Tensor Cores' do not support 32-bit (aka: 23-bit mantissa) or higher.

Yup, in a semi-related field, NVIDIA has 3xTF32 for cases needing higher precision: https://github.com/NVIDIA/cutlass/discussions/361

There's a paper on getting fp32 accuracy using tf32 tensor cores and losing 3x efficiency. Can't wait to try it with cutlass... once I get how to use cutlass, woof.

The catch is it's only for TF32 computations (Nvidia proprietary 19 bit floating point number format)

I missed that, to me that makes the ‘32’ in the name misleading.

TF32 = FP32 range + FP16 precision

Why not call it TF19 then.

Because your existing FP32 models should run fine when converted to TF32, so TF32 is equivalent to FP32 as far as DL practitioners are concerned.

There is a lot of redundancy in DL that forgives all manner of sins; think it’s sneaky.

Because it's 32-bits wide in memory.

The effective mantissa is like FP16 but it's padded out to be the same size as FP32.

In other words, there's 1 sign bit, 8 exponent bits, 10 mantissa bits that are USED, and 13 mantissa bits that are IGNORED.

1 + 8 + 10 + 13 = 32

The 13 ignored mantissa bits are part of the memory image: they pad the number out to 32-bit alignment.

But the user never sees that memory right? Doesn't it go in FP32 and come out FP32? I still think it's deceptive marketing.

The user does see 32-bits and all bits are used because all the additions (and other operations besides the multiply in matrix ops) are in FP32. So the bottom bits are populated with useful information.

And with 50% sparsity

DP Linpack flops is what counts in supercomputer ranking. Stuck at .44 Exoflops in 2021.

700 watts so being NVidia it'll blow up in 6 months and you'll need to wait in a queue for 6 months to RMA it because all the miners had bought up the entire supply chain.

Those datacenter/hpc GPUs don't seem to get bought so much by the mining community? I don't have problems sourcing some through the usual channels (HPE, dell,...?). But you need somehow deep pockets.

Given that it's Nvidia, no Linux support. That's the catch.

I thought that only applied to their consumer products.

Their consumer products have Linux support too, the catch is just that the drivers are proprietary binary blobs

Don't they provide Linux drivers for their gaming graphics cards too, just not open source?

Yes

No Linux support? Guess I'll have to keep using Solaris with my A4000!

Nvidia provides Linux drivers for their server chips.

Don't they provide them for their consumer cards too, just that it's a closed source binary blob?

And not just Linux: FreeBSD.

* https://www.nvidia.com/en-us/drivers/unix/freebsd-x64-archiv...

* https://www.freshports.org/x11/nvidia-driver

Heck, Solaris:

* https://www.nvidia.com/en-us/drivers/unix/solaris-display-ar...

* https://www.nvidia.com/en-us/drivers/unix/

CUDA and related software/libraries only work on Linux or Windows.

Some are even Linux only like nccl (AFAIK required to fully use NVLink)

That's a strange statement. The vast vast majority of these cards will be in systems running Linux.

I for one suffer deeply when I try to install the nvidia drivers on Linux. The website binaries _always_ break my system

Only the ppas from graphics-drivers work properly

My experience on windows is much more automatic and it never breaks anything. But I'd rather pay the price (installing on Linux) to avoid windows at all costs

Yea it's not ideal but really no option is. Built in to Linux would be a problem too given the rate of GPU driver development. Most Linux installs in the corporate world are stuck on the major version of the kernel and system packages they shipped with.

If you installed the drivers using the PPAs, you can't then update using the NVIDIA-provided binaries without doing a very thorough purge, including deleting all dependent installs (CUDNN, CUBLAS, etc.)

I highly recommend sticking with one technique or the other; never intermix them.

All the AI software running on these data-centre chips is almost exclusively running on Linux.

I wish people would stop talking rubbish about NVIDIA's Linux support.

That's because nvidias linux support for consumers is indeed trash, while their creators/business/creatives software (eg CUDA) is not trash, but you mostly hear consumers trashing nvidia.

They don't make (relevant) money from consumer hardware on Linux.

Only FOSS zealots actually, the rest of us is quite ok with their binary drivers.

Reply on news.ycombinator.com