I'm sure they've done benchmarking comparisons with the A100 and H100, but since they're not sharing them it seems unlikely they show the TPU's in a favorable light. Plus, they pick up more spend every time someone runs these benchmarks for themselves.

The most important reason all the Big Tech companies spin their own AI chips is that NVIDIA margins are so insane. A purchase from NVIDIA at the hyperscaler volume is on the same order of magnitude as spinning your own chip.

Generally TPUs don't beat Nvidia per chip. The real reason Google likes them is they are much cheaper in TCO for Google. Of course they may not decide to pass on all the savings to you.

Indirectly, using recent 50k tpuv5e run info [1], I'd guess it will give 50-70% of H100 in large-scale LLM jobs: MLCommons show that H100 give a bit more than 500 TFLOPs in FP8, and v5e gives 100 TOPs in INT8. v5p has 2.3 times more theoretical OPs, 3 times more interchip bandwidth and 3.3 times more memory bandwidth, so assuming you can extrapolate and given that bandwidth is usually a bottleneck, ~60% seems plausible.

[1] https://cloud.google.com/blog/products/compute/the-worlds-la...

[2] https://github.com/mlcommons/training_results_v3.1

v4 chips are basically the same in bf16 performance as A100(slower in int8).