Google announces TPU v5p with specs competing against H100 when NVIDIA GH200 Grace Hopper is announced.
>The new pods also have 95GB of high-bandwidth memory (HBM)
GH200 has 282GB of HBM3e memory
>These new pods provide a throughput of 4,800Gbps.
That equals 600 GB/sec. GH200 has 10TB/sec of combined bandwidth HBM3e memory.
There are also a few things wrong with the specs you quoted. 282 GB is split between two GPUs, it's 144 GB per GPU (not sure where the extra 6 GB went). The TPU pod throughput number you quoted is interconnect bandwidth which you compared to GH200's memory bandwidth. Those numbers are not comparable.
I believe GH200 does beat TPU v5p per chip, however the differences are not anywhere near as large as your comparison suggested. And it's likely that TPU v5p is dramatically more cost effective but we don't have Google's internal numbers to prove that.
There is only one GPU in GH200 and it gets all 282 GB HBM3e memory. There is another 480 GB LPDDR5X memory for the another chip (Grace CPU, not GPU).
GH200 interconnect bandwidth trough NVLink is 900 GB/s.
The 282 GB figure comes from a "dual configuration" server with two CPU chips and two GPU chips:
> the new platform will be available in a wide range of configurations. The dual configuration [...] comprises a single server with 144 Arm Neoverse cores, eight petaflops of AI performance and 282GB of the latest HBM3e memory technology.
Note that one Grace Hopper CPU has 72 cores so this configuration clearly has two CPUs and two GPUs in one server, and the 282 GB HBM3e total is split between the two GPUs with 144 GB each (again, not sure where the extra 6 GB went, maybe disabled for yield issues?).
https://nvidianews.nvidia.com/news/gh200-grace-hopper-superc...
Nothing would be worse than a massive hardware recall.
If anything, Google would have an easier time going live sooner than Nvidia since they own all the TPUs and they're onsite.
> Google's v5p TPUs are up to 2.8 times faster at training large language models than TPU v4, and offer 2.1-times value-for-money.
That's a comparison of TPUv5p to TPUv4, not to H100.
One Grace Hopper has: H100 chip, Grace CPU with 72 cores, 282GB of HBM3e memory and 480 GB LPDDR5X for the CPU.
Actually, DGX GH200 seem to come in different sizes, something that I can't find clearly stated on Nvidia's website, but what I do see are entirely inconsistent specs. I'm thinking they've changed it a few times.
https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20... describes the DGX GH200 as:
-256 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU)
-each Grace CPU has 480 GB LPDDR5 CPU memory
-each H100 GPU has 96 GB of HBM3
-each GPU can access all of the CPU and GPU memory in the whole system (144TB), at 900 GBps
At https://www.nvidia.com/en-us/data-center/dgx-gh200/ (linked to from the DGX page from the website) which has a Datasheet pdf they say the DGX GH200 has:
-32 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU)
-each Grace CPU has 72 "ARM Neoverse V2 Cores with SVE2 4X 128"
-each GPU can access "19.5TB" shared memory
-that's 624GB per superchip, which is weird. I expect it's actually 96GiB HBM3 + 512GiB LPDDR, a total of 19456GiB = 19.0TiB
And other people have found completely different specs elsewhere on the website!
Each TPUv5p host has 208 vCPU, 448 GB ram, 200 Gbps NIC (for data loading), 4 TPUv5p chips (8 cores). Each TPU chip has 95GB HBM with 2765GBps memory bandwidth, 3D interconnect with 4800 Gbps (3D interconnect means every chip has 6x 800 Gbps feeds, each going to a different chip in the torus).
One big difference between TPUs and GPUs is that the former have few beefy cores (2x chips) while the latter a lot of smaller cores. It can help or hinder depending on the workload. It makes it simpler to do deterministic training, which helps both debugging and optimization.
So, each system has 448GB host RAM and 4 TPU chips with 380GB HBM. The amount of host memory doesn't really matter that much: it's there to make sure that accelerators can be fed data at full speed.
TPU5p pods are organized in "cubes" of 16 machines (64 TPUs) that can be assembled in any shape for different kinds of data/model parallelism (see twisted tori in the TPUv4 paper). Each cube has 6080GB (6TB) of HBM ram and 29.3 petaflops (bfloat16) of theoretical peak performance. You can get multiple exaflops from a single pod and use multiple pods in the same datacenter with multislice if needed.
TPU are more power efficient than GPUs (e.g. A100 has a TDP 3x of TPUv4 + 3D torus requires fewer connections that a fattree interconnect, and optics do consume their fair bit of power), so you can scale to higher compute per datacenter before you hit that bottleneck on really really large models.
Also notice that you are using bare metal numbers for Grace hopper and I can only quote (what is publicly available) userspace/VM numbers for TPUs. For example, the actual physical RAM on the hosts is higher than 448 GB; so 448 vs 480 is apples to oranges.
Source: https://cloud.google.com/tpu/docs/v5p-training + I am oncall for these platforms.
https://arxiv.org/ftp/arxiv/papers/2304/2304.01433.pdf for TPUv4 vs. A100
---
I believe Grace Hopper is an amazing platform. It is a more general one than TPUv5p pods. Each optimized for different constraints and TCO/perf.
>Each NVIDIA Grace Hopper Superchip in NVIDIA DGX GH200 has 480 GB LPDDR5 CPU memory, at an eighth of the power per GB, compared with DDR5, and 96 GB of fast HBM3.
(https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20...)
Looks like a single superchip has 96GB of hbm3, similar to a single v5p tpu. The H100's in the DGX H100 were 80GB per board.