This is for Google Cloud users. My understanding is that Google mostly uses TPU internally.

Back of the envelope math is that H100 is twice as fast as A100 (task may vary). So your 1000-node A100 very, very fast.

Now, the GPU-to-GPU links (NVLink) might often give them a big advantage for some workloads, letting them exchange data without going through the CPU, and virtually address more memory if your want to manipulate very large models.

So it's hard to answer properly without knowing the topology of your cluster.

Also, note that this "supercomputer", is probably "just" a DGX H100 in Google's DC.

They use their own TPUs like described in this paper [0]. They talk about 4096-chip supercomputers so this should give you an idea about what we are talking here. The paper is pretty fascinating stuff. They are using optical interconnects for example, which sounded like science fiction a few years ago.

[0] https://arxiv.org/pdf/2304.01433.pdf