Hacker News Reader

Comment by bushbaba | Hacker News Reader

Does this mean google just deprecated TPUs? Not surprised.

> Does this mean google just deprecated TPUs?

No, it is the 9,163,584th [0] indication that Google likes to pursue multiple solutions in the same space in parallel with different submarkets, risk profiles, expected payoff terms, or other dimensions.

[0] this is a conservative estimate

this looks like it's for GCP. TPUs are used for most internal workloads. It's available externally but some of the papercuts and devex without the TPU/TF team helping you can be more painful than using Nvidia/CUDA

TPUs do compete with GPUs for ML tasks, so yes, this is evidence that GPUs are winning.

The only alternative I could imagine is that TPUs will "win" at supercomputers exclusivity aimed at inference (as opposed to training). Since TPUs excel at inference. The question is how much ML compute is used for inference as opposed to training. Not much, I guess, otherwise something like TPUs would be more popular.

I've heard estimates that the amount of compute used to train GPT-4 is equivalent to 8 months of usage and most models are used much less than GPT-4 is, although I guess they are also easier to train.

of "usage"? I never bought that claim as it's not clear what usage they mean – on 100x8 months or 10000x8 months?

Since you seem knowledgable on this topic, what is it that TPUs do differently than GPUs? Why are they better at inference?

They have published various papers and technical reports. The main aim is to make them inhouse and more efficient. Each generation is a little different, like (iirc) v3 is not for training, more for serving at inference time. The use different floating point format and circuits, so they are not good for scientific workloads, iirc again.

Sorry, I actually don't know much about them.

Why would you assume that?

Either GPUs are better for most AI tasks or TPUs. Both being overall approximately equally good is very unlikely.

> most AI tasks

Different workloads require different infrastructure.

Can your workload saturate the TPU without getting throttled by memory or network? Great! Use TPUs and reduce training cost.

But if your TPUs are idle 70% of the time because the constraint is getting data to them ...

"A3 represents the first production-level deployment of its GPU-to-GPU data interface, which allows for sharing data at 200 Gbps while bypassing the host CPU. This interface, which Google calls the Infrastructure Processing Unit (IPU), results in a 10x uplift in available network bandwidth for A3 virtual machines (VM) compared to A2 VMs."

TPU's have a TPU-TPU interconnect that is faster and lower latency than any GPU cluster [1]. That said this is a huge leap for GPU's on GCP. For A100's SOTA is 1.6tbit per host over Infiniband (which azure and some smaller gpu clouds provided), AWS had 400-800 Gbit and GCP had .... ~100gbit.

SOTA seems to be 3.2Tbit for H100 clusters so this still seems a bit slow? (Tricky as they don't give us a clear number just 10x). H100's are much more powerful per chip though so at least initially the clusters will be smaller and not network bound.

The tricky thing is no one other than Azure of the big providers seems willing to pay Nvidia's margins for RDMA switches, it seems this is still the case.

[1] https://arxiv.org/pdf/2304.01433.pdf

But you could do an equivalent TPU<>TPU interlink. Surely that can’t be the reason.

That's why I said "most" and "overall". Of course TPUs will have a niche. But it looks like the vast majority of money spent on ML compute is converging on GPUs.

Hmm no?

Clouds offer many competing offerings because different clients have different needs.

Reply on news.ycombinator.com