Google TPU v5p beats Nvidia H100
techradar.com
techradar.com
The thing is that Wall St doesn't understand any of the technical details and will just see huge profit growth from NVDA and drive the price crazy high.
1. Nvidia GPUs are dominant at training
2. Inference is easier than training, so other cards will become competitive in inference performance.
3. As AI applications start to proliferate, inference costs will start to dominate training costs.
4. Hence Nvidia's dominance will not last.
I think the most problematic assumption is 3. Every AI company we see thus far is locked in an arms race to improve model performance. Getting overtaken by another company's model is very harmful for business performance (See Midjourney's reddit activity after DALLE-3), while a SOTA release instantly results in large revenue leaps.
We also haven't reached the stage where most large companies can fine-tune their own models, given the sheer complexity of engineering involved. But this will be solved with a better ecosystem, and will then trigger a boom in training demand that does scale with the number of users.
Will this hold in the future? Not indefinitely, but I don't see this ending in say 5 years. We are far from AGI, so scaling laws + market competition mean training runs will grow just as fast as inference costs.
Also, 4 is very questionable. Nvidia's cards are not inherently disadvantaged in inference, they may not be specialized ASICs, but are good enough for the job with an extremely mature ecosystem. The only reason why other cards can be competitive against Nvidia's cards in inference, is because of Nvidia's 70% margins.
Therefore, all Nvidia needs, to defend against attackers, is to lower their margins. They'll still be extremely profitable, their competitors not so much. This is already showing in the A100 H100 bifurcation. H100s are used for training, while the now old A100s used for inference. Inference card providers will need to compete against a permanently large stock of retired Nvidia training cards.
Apple is still utterly dominant in the phone business after nearly 2 decades. They capture the majority of the profits despite
1. Not manufacturing their own hardware
2. The majority of the market share by units sold is say Chinese/Korean
If inference is easy, while training is hard. It could just lead to Nvidia capturing all the prestigious and easy profits from training, while the inference market is a brutal low margin business with 10 competitors. This will lead to the Apple situation.
I think Nvidia will continue to be dominant because it's still a lot easier to go from Cuda training to Cuda (TensorRT acerbated lets say) inference than migrating your model to ONNX to get it to run on some weird inference stack.
Because you want to learn a new gait on your robot in a few minutes.
What stats are you looking at? Looking at https://subredditstats.com/r/midjourney , I see a slower growth curve after the end of July, but still growing and seemingly unrelated to the DALL-E 3 release, which was more like end of October publicly.
Do you mind expanding on this? What do you see as the biggest things that make that milestone > 5 years away?
Not trolling, just genuinely curious -- I'm a distributed systems engineer (read: dinosaur) who's been stuck in a dead-end job without much time to learn about all this new AI stuff. From a distance, it really looks like a time of rapid and compounding growth curves.
Relatedly, it also does look -- again, naively and from a distance -- like an "AI is going to eat the world" moment, in the sense that current AI systems seem good enough to apply to a whole host of use cases. It seems like there's money sitting around just waiting to be picked up in all different industries by startups who'll train domain-specific models using current AI technologies.
In reality, your gut reaction is correct. We have turned general intelligence into a commodity. Any aspect of any situation, process, system, domain, etc. which was formerly starved of intelligence may now be supplied with it. The value unlock here is unspeakable. The possibilities are so vast that many of us fill with anxiety at the thought.
It may be true that inference is 100x larger than training in terms of raw compute. But I think it very well could be that inference is only 10x larger, or same-sized.
And besides, you can look at it in terms of sheer inputs and outputs. The size of data yet to be trained on is absolutely enormous. Photos and video, multimedia. Absolutely enormous. Hell, we need giant stacks of H100's for text. Text! The most compact possible format!
> Unlike Nvidia, which offers its GPUs out for other companies to purchase,
> Google's custom-made TPUs remain in-house for use across its own products and services.
nobody can purchase one of these.
and even if someone external to google could purchase one, why would they trust google for software, documentation or (the big if with google) ongoing support. this isn't their core business.
nVIDIA's hardware offerings are products for sale. Google's TPUs are a value-add to their rental market. This distinction is not lost on people. There's a reason all Google's big-load TPU clients they list in press releases are companies Google invested in.
Google announces TPU v5p with specs competing against H100 when NVIDIA GH200 Grace Hopper is announced.
>The new pods also have 95GB of high-bandwidth memory (HBM)
GH200 has 282GB of HBM3e memory
>These new pods provide a throughput of 4,800Gbps.
That equals 600 GB/sec. GH200 has 10TB/sec of combined bandwidth HBM3e memory.
> Google's v5p TPUs are up to 2.8 times faster at training large language models than TPU v4, and offer 2.1-times value-for-money.
That's a comparison of TPUv5p to TPUv4, not to H100.
One Grace Hopper has: H100 chip, Grace CPU with 72 cores, 282GB of HBM3e memory and 480 GB LPDDR5X for the CPU.
Actually, DGX GH200 seem to come in different sizes, something that I can't find clearly stated on Nvidia's website, but what I do see are entirely inconsistent specs. I'm thinking they've changed it a few times.
https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20... describes the DGX GH200 as:
-256 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU)
-each Grace CPU has 480 GB LPDDR5 CPU memory
-each H100 GPU has 96 GB of HBM3
-each GPU can access all of the CPU and GPU memory in the whole system (144TB), at 900 GBps
At https://www.nvidia.com/en-us/data-center/dgx-gh200/ (linked to from the DGX page from the website) which has a Datasheet pdf they say the DGX GH200 has:
-32 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU)
-each Grace CPU has 72 "ARM Neoverse V2 Cores with SVE2 4X 128"
-each GPU can access "19.5TB" shared memory
-that's 624GB per superchip, which is weird. I expect it's actually 96GiB HBM3 + 512GiB LPDDR, a total of 19456GiB = 19.0TiB
And other people have found completely different specs elsewhere on the website!
Each TPUv5p host has 208 vCPU, 448 GB ram, 200 Gbps NIC (for data loading), 4 TPUv5p chips (8 cores). Each TPU chip has 95GB HBM with 2765GBps memory bandwidth, 3D interconnect with 4800 Gbps (3D interconnect means every chip has 6x 800 Gbps feeds, each going to a different chip in the torus).
One big difference between TPUs and GPUs is that the former have few beefy cores (2x chips) while the latter a lot of smaller cores. It can help or hinder depending on the workload. It makes it simpler to do deterministic training, which helps both debugging and optimization.
So, each system has 448GB host RAM and 4 TPU chips with 380GB HBM. The amount of host memory doesn't really matter that much: it's there to make sure that accelerators can be fed data at full speed.
TPU5p pods are organized in "cubes" of 16 machines (64 TPUs) that can be assembled in any shape for different kinds of data/model parallelism (see twisted tori in the TPUv4 paper). Each cube has 6080GB (6TB) of HBM ram and 29.3 petaflops (bfloat16) of theoretical peak performance. You can get multiple exaflops from a single pod and use multiple pods in the same datacenter with multislice if needed.
TPU are more power efficient than GPUs (e.g. A100 has a TDP 3x of TPUv4 + 3D torus requires fewer connections that a fattree interconnect, and optics do consume their fair bit of power), so you can scale to higher compute per datacenter before you hit that bottleneck on really really large models.
Also notice that you are using bare metal numbers for Grace hopper and I can only quote (what is publicly available) userspace/VM numbers for TPUs. For example, the actual physical RAM on the hosts is higher than 448 GB; so 448 vs 480 is apples to oranges.
Source: https://cloud.google.com/tpu/docs/v5p-training + I am oncall for these platforms.
https://arxiv.org/ftp/arxiv/papers/2304/2304.01433.pdf for TPUv4 vs. A100
---
I believe Grace Hopper is an amazing platform. It is a more general one than TPUv5p pods. Each optimized for different constraints and TCO/perf.
There are also a few things wrong with the specs you quoted. 282 GB is split between two GPUs, it's 144 GB per GPU (not sure where the extra 6 GB went). The TPU pod throughput number you quoted is interconnect bandwidth which you compared to GH200's memory bandwidth. Those numbers are not comparable.
I believe GH200 does beat TPU v5p per chip, however the differences are not anywhere near as large as your comparison suggested. And it's likely that TPU v5p is dramatically more cost effective but we don't have Google's internal numbers to prove that.
There is only one GPU in GH200 and it gets all 282 GB HBM3e memory. There is another 480 GB LPDDR5X memory for the another chip (Grace CPU, not GPU).
GH200 interconnect bandwidth trough NVLink is 900 GB/s.
The 282 GB figure comes from a "dual configuration" server with two CPU chips and two GPU chips:
> the new platform will be available in a wide range of configurations. The dual configuration [...] comprises a single server with 144 Arm Neoverse cores, eight petaflops of AI performance and 282GB of the latest HBM3e memory technology.
Note that one Grace Hopper CPU has 72 cores so this configuration clearly has two CPUs and two GPUs in one server, and the 282 GB HBM3e total is split between the two GPUs with 144 GB each (again, not sure where the extra 6 GB went, maybe disabled for yield issues?).
https://nvidianews.nvidia.com/news/gh200-grace-hopper-superc...
Nothing would be worse than a massive hardware recall.
If anything, Google would have an easier time going live sooner than Nvidia since they own all the TPUs and they're onsite.
>Each NVIDIA Grace Hopper Superchip in NVIDIA DGX GH200 has 480 GB LPDDR5 CPU memory, at an eighth of the power per GB, compared with DDR5, and 96 GB of fast HBM3.
(https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20...)
Looks like a single superchip has 96GB of hbm3, similar to a single v5p tpu. The H100's in the DGX H100 were 80GB per board.
The media keeps missing the real lock in Nvidia has: CUDA. It's not the hardware. It's the ability for someone to use it painlessly.
Additionally many operations that run on GPU but are just unsupported for TPU. Sparse tensors have pretty limited support and there's bunch of models that will crash on TPU and require refactoring. Sometimes pretty heavy thousands of lines refactoring.
edit: Pytorch is even worse. Pytorch does not implement efficient tpu device data loading and generally has poor performance no where comparable to tensorflow/jax numbers. I'm unaware of any pytorch benchmarks where tpu actually wins. For tensorflow/jax if you can get it running and your model suits tpu assumptions (so basic CNN) then yes it can be cost effective. For pytorch even simple cases tend to lose.
Unless you physically work next to the TPU hardware team, the torch support for TPUs is pretty brittle.
of course not ideal, but better than "open sourcing" it and refusing every request because it does not work for their codebase. worse than having it open source from the get go, of course.
assuming that day comes, does it have a competitor in the works? a python superset, compatible with python libs, but enables you to go bare metal to the point that it enables you to directly program GPUs and TPUs without CUDA or anything?
"never" means you believe it will never be open sourced, or a competitor will surpass it by the time it is open sourced. or that you believe the premise of the lang is flawed and we don't need such a thing. Which one is it?
Here is their github btw: https://github.com/modularml/mojo
From what I see, they have a pretty active community and there is demand for such a system.
The github says something similar:
>This repo is the beginning of our Mojo open source effort. We've started with Mojo code examples and documentation, and we'll add the Mojo standard library as soon as we get the necessary infrastructure in place. The challenge is that we use Mojo pervasively inside Modular and we need to make sure that community contributions can proceed smoothly with good build and testing tools that will allow this repo to become the source of truth (right now it is not). We'll progressively add the necessary components, such as continuous integration, build tools, and more source code over time.
[Citation needed]
The creator, Chris Lattner, previously created LLVM, clang, and Swift. In each case he said these projects would be open sourced, and in each case they were. In each case they reached mainstream adoption in their respective target markets.
He's stated that Mojo will be open source.
If you're going to claim with great confidence that this language will have a different outcome to his previous ones, then you probably should have some strong evidence for that.
The TPU hardware is great in a lot of ways and it allowed google to move quickly in ML research and product deployments, but I don't think it was ever a money-maker for cloud.
- Google search (vs previous entrenched search engines in the early '00s)
- Adsense/doubleclick (vs early ad networks at the time)
- Gmail (vs aol, hotmail, etc)
- Android (vs iOS, palm, etc)
- Chrome (vs all other browsers)
Sure, i'm picking the obvious winners, but these are all market leaders now (Android by global share) where earlier incumbents were big, but not Google-big.
Even if Google's use of TPUs are purely self-serving, it will have a noticeable effect on their ability to scale their consumer AI usage at diminishing costs. Their ability to scale AI inference to meet "Google scale" demand, and do it cheaply (at least by industry standards), will make them formidable in the "ai race". This is why altman/microsoft and others are investing heavily in AI chips.
But I don't think their TPU will be only self-serving, rather, they'll scale it's use through GCP for enterprise customers to run AI. Microsoft is already tapping their enterprise customers for this new "product". But those kinds of customers will care more about cost than anything else.
The long-term game here is a cost game, and Google is very, very good at that and has a headstart on the chip side.
Really? What if someone writes a new back-end to PyTorch, TensorFlow and perhaps a few other popular libraries? Then will CUDA still matter that much?
If that was easy to do surely AMD would have done it by now? After many years of trying?
* Whether something 'beats' something else is actually a question of price/performance and...
* Whether or not the vendor in question is famous for dropping people in the shit.
And by "performance" I mean the quality of the end result, the training speed, and what features device is physically capable of handling. While kinda discrete aspects in other fields, all of these are highly correlated in ML training.
One specific example I am thinking of is support for efficient long context training. If your stack doesn't, for instance, support flash attention (or roll its own custom implementation), then good luck with that.
Another I am thinking of is quantization. Sometimes you want to train with a specific quantization scheme, and sometimes doing that is just a performance compromise. And vendors really love to fudge their numbers with the fastest, most primitive quantization on the device.
Compatability - does using a tpu require reworking significant parts of your software stack and application stack? If so, that sucks for most companies who's researchers are used to using nvidia libraries, tooling and hardware to get their models running, and reworking the entire bottom end of that to work on an esoteric platform needs to equate to a huge cost savings at scale to be worth it.
Cost per tensor flop delivered - likely very low if google has optimized the silicon, memory, voltage, power and temperature envelope, networking, boards and chassis for the server running it, as well as optimizing for optimal process node efficiency/cost. They're probably not on bleeding edge tsmc process, but instead optimizing for total deployed running cost per pflop over 2-3 years.
It's also now public (as of November) that microsoft/azure have been working for many years on their own ai chip, dubbed "Maia" [1], and appliance with the obvious goal of taking some of that nvidia margin in-house (and with openai/bing/copilot being a massive consumer of capacity). I think this will become even more commonplace with cloud vendors - even medium size ones - than it already is. The knowledge and complexity barrier to designing a tile processor unit seems pretty low and it looks like most of the hard stuff is in the drivers and software - something cloud providers designing and integrating internally can bypass and control to a great degree.
It's also very hard to benchmark side by side on these since I'm sure cuda/nvidia hardware can do compute that a TPU cannot. AMD's machine learning accelerators look good on paper too, as do the tensor processors in the apple silicon, but on real world applications and use cases they don't often measure up save for a few optimal workflows.
[1] https://www.geekwire.com/2023/microsoft-unveils-custom-ai-ch...
H100s are less stable training though.
V5ps are about the same as intel gaudi2s and both are cheaper than h100s price/perf
Gaudi3s are 3-4x faster…
Also, how Gaudi3 compares to Google TPUs? Thanks! I know that Gaudi3 is not readily available yet.
Meanwhile I am still waiting for Tesla Dojo to be an actual thing...
I don't recall if they made comments about public availability for Dojo before, but given the difficulty of competing in this market and the huge benefits scale brings in hardware, it would seem foolish to limit the system to internal customers only.
Maybe they shouldn’t have gone with an in-house design.
> it would seem foolish to limit the system to internal customers only.
My understanding from working with similar HPC systems is that public use cases are likely niche and many tasks don’t benefit from the sort of fast interconnect their setup gives them. If anything being quite public about their design was meant to allow other companies to copy their design and compete with nvidia on a united front perhaps.
It’s not really as versatile as AWS, GCP are to profitably work as a compute cloud provider though. Perhaps it could be a provider for other companies who don’t want to own their hardware to do training runs on.
But Google can build good framework integration. Will wait and see.
Could not find Gaudi or Cerebras device available anywhere. And MI300 risky since software support is poor. But if you have managed to make it work, perhaps there's potential.
What stack are you using and what workload?
https://arxiv.org/pdf/2309.07181.pdf
Meaning the TPU v5p is likely slower than H100 for most ML workloads that depend on pytorch.
Imagine a future where there are many players in the cloud training business. These players have the chance to retrieve de LLM model and since this could be a high stake business that is dangerous. I underastand that currently cloud players can observe everything that happens in their clouds including secret information and this is not happenning but it could not be "Google" by themselves but an inside job.
There's still work to do -- lots of repositories still contain `if device=="cuda"` type language, but my own experience is that manging code around to use apple gpus has gotten vastly easier this year, and I see more and more AMD GPU owners floating around github issues with resolvable problems. A year ago they were barely present.
All that said - people aren't ignoring it - and entrepreneurs and cloud monopolies are putting real resources into opening things up. I think the playing field will continue to level / get back to competitive over the next two to three years.
And while many problems are trivial to fix at eg the PyTorch layer, lots of stuff like flash attention, DeepSpeed, etc. are coded directly in CUDA kernels.
How is CUDA so sticky when most ML devs aren't writing CUDA but something several layers of abstraction above it? Why can't intel, AMD, google w/e come along and write an adapter for that lowest level to TF, pytorch or whatever is the framework of the day?
A long long time ago, i.e. the last time AMD was competing with Intel (before this time, that is), we used to use Intel's icc in our lab to optimize for the Intel CPUs and squeeze as much as possible out of them. Then AMD came out with their "Athlon"(?) and it was an Intel-beater at that time. But AMD never released a compiler for it; I bet they had one internally, but we had to rely on plain old GCC.
These hardware companies don't seem to get that a kick-ass software can really add wings to their hardware sales. If I were a hardware vendor, I would, if nothing else, make my hardware's software open so the community can run with it and create better software; which will result in more hardware sales!
This is a really tricky guess, case in point that AMD's latest chip cant compete on training because they could not get Flash Attention 2 working on the backward pass because of their hardware architecture. [1]
Attempts to abstract at a higher layer have failed so far because that lower layer is really valuable, again Flash Attention is a good example.
[1] https://www.semianalysis.com/p/amd-mi300-performance-faster-...
The API that matters is Torch, and only the API. Letting NVIDIA charge famine prices is both a bad idea and a huge incentive to write software bridging the gap.
The amount of extremely CUDA specific handwritten compute architecture code, custom kernels, etc has exploded and other than a few things here and there (yes, like torch SDPA) we’re waaaay past vanilla torch for anything beyond a toy.
The pricing and theoretical hardware specs of these novelties doesn’t matter when an inferior on paper Nvidia product will wipe the floor with the other in the real world.
People have spent 15 years wringing every last penny out of CUDA on Nvidia hardware.
There is some light shining through but you’ll still see things like “Woo-hoo FlashAttention finally supports ROCm! Oh wait, what’s that, FlashAttention2 has been running on CUDA for six months?”
Don’t even get me started on the “alternative” software stacks and drivers.
The graphical debugging tools that allows stepping through GPU code just like with on the CPU.
This is why everyone is trying to compete with CUDA. Everyone wants a slice of the pie as we go from tiny amount to everything.