Hacker News Reader

Comment by throwaway_ab | Hacker News Reader

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia?

If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone.

From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific GPU's.

View on HN

Nvidia has decades of experience selling hardware to people with all the pains that entails, support, sales channels, customer acquisition, software, it's something you don't just do overnight, and it does cost money. Google's TPUs get some of their cost efficiency from not supporting COTS use cases and the overhead of selling to people, and the total wall clock time has to also include the total operational costs, which dominate at their size (e.g. if it's 30x slower but 1/50th the TCO then it's a win. I don't know how TPUv5 stacks up against the B200). It's not as simple as "just put it on a shelf and sell it and make a gajillion dollars like nvidia"

TPU v5p is ~2 times slower than H100 at larg(ish)-scale training (order of 10k chips) [1]. And they already have v6 [2]. I think it's safe to say that they are fairly close to Nvidia in terms of performance.

[1] https://mlcommons.org/benchmarks/training/

[2] https://cloud.google.com/blog/products/compute/introducing-t...

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

So you're saying, H100s are the corporate equivalent of Raspberry Pis now? Bought to not miss out, then left to gather dust in a drawer?

Hello! If you're interested in monetizing those GPUs, I'd be happy to rent them (all 400!) and offer those to customers of the cloud I work at :)

jonathan [at] tensordock.com

If you want 512 H100s connected with infiniband: https://lambdalabs.com/service/gpu-cloud/1-click-clusters

Have you considered sponsoring an open-source project? ;)

If you'd like to contribute some much-appreciated compute to university researchers all across the US, please email me (mhsiu at ucsd.edu).

It should be possible to hook up your idle devices to Nautilus (https://nationalresearchplatform.org/), which is a cluster set up to support researchers at a bunch of universities. I can't guarantee anything since I'm not involved in the cluster management itself, but I can put you in contact with those who are if you're interested.

I know of many important projects that need GPUs right now and aren’t getting any. You could help motivate the ponydiffusion folks to actually try finetuning of SD3!

Wow, that's a lot of money in inventory. What was the original thought process? Just fomo?

the world would love to buy time on your idle H100's if you're selling.

That's insane and incredible all at the same time.

Looking to get rid of a few?....

Probably could profit selling them second hand honestly.

They do sell shovels, you can get Google TPUs on Google Cloud.

Exactly and they are still about 1/18ths as good at training llms as a H100.

Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.

Given that Google invented Transformer architecture (and Google AI continues to do foundational R&D on ML architecture) — and that Google's TPUs don't even support the most common ML standards, but require their own training and inference frameworks — I would assume that "the point" of TPUs from Google's perspective, has less to do with running LLMs, and more to do with running weird experimental custom model architectures that don't even exist as journal papers yet.

I would bet money that TPUs are at least better at doing AI research than anything Nvidia will sell you. That alone might be enough for Google to keep getting some new ones fabbed each year. The TPUs you can rent on Google Cloud might very well just be hardware requisitioned by the AI team, for the AI team, that they aren't always using to capacity, and so is "earning out" its CapEx through public rentals.

TPUs are maybe also better at other things Google does internally, too. Running inference on YouTube's audio+video-input timecoded-captions-output model, say.

If you're interested in a peer reviewed scientific comparison, Google writes retrospective papers after contemporary TPUs and GPUs are deployed versus speculation about future products. The most recent compares TPU v4 and A100. (TPU v5 and H100 is for a future paper). Here is a quote from the abstract:

"Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. ... For similar sized systems, it is ~4.3x--4.5x faster than the Graphcore IPU Bow and is 1.2x--1.7x faster and uses 1.3x--1.9x less power than the Nvidia A100. TPU v4s inside the energy-optimized warehouse scale computers of Google Cloud use ~2--6x less energy and produce ~20x less CO2e than contemporary DSAs in typical on-premise data centers."

Here is a link to the paper: https://dl.acm.org/doi/pdf/10.1145/3579371.3589350

That quote is referring to the A100... the H100 used ~75% more power to deliver "up to 9x faster AI training and up to 30x faster AI inference speedups on large language models compared to the prior generation A100."[0]

Which sure makes the H100 sound both faster and more efficient (per unit of compute) than the TPU v4, given what was in your quote. I don't think your quote does anything to support the position that TPUs are noticeably better than Nvidia's offerings for this task.

Complicating this is that the TPU v5 generation has already come out, and the Nvidia B100 generation is imminent within a couple of months. (So, no, a comparison of TPUv5 to H100 isn't for a future paper... that future paper should be comparing TPUv5 to B100, not H100.)

[0]: https://developer.nvidia.com/blog/nvidia-hopper-architecture...

As someone unfamiliar with this area, can one of the downvotes explain why they choose to downvote this? Is it wrong?

I'm sure it probably is faster for thier own workloads (which they are choosing to benchmark on), why bother making it if not. But that is clearly not universally true, a GPU is clearly more versatile. This means nothing to most if they can't for example train an LLM on them.

I don't see how you can evaluate better and worse for training without doing so on cost basis. If it costs less and eventually finishes then it's better.

This assumes that you can linearly scale up the number of TPUs to get equal performance to Nvidia cards for less cost. Like most things distributed, this is unlikely to be the case.

This is absolutely the case, TPUs scale very well: https://github.com/google/maxtext .

The repo mentiones a Karpathy tweet from Jan 2023. Andrej has recently created llm.c and the same model trained about 32x faster on the same NVidia hardware mentioned in the tweet. I dont think the perfomance estimate that the repo used (based on that early tweet) was accurate for the performance of the NVidia hardware itself.

Time is money. You might be a lab with long queues to train, leaving expensive staff twiddling their thumbs.

Also energy cost. 18 chips vs 1, it's probably costing a lot more to run 18

Google claims the opposite in "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings " https://arxiv.org/abs/2304.01433

Despite various details I don't think that this is an area where Facebook is very different from Google. Both have terrifying amounts of datacenter to play with. Both have long experience making reliable products out of unreliable subsystems. Both have innovative orchestration and storage stacks. Meta hasn't published much or anything about things like reconfigurable optical switches, but that doesn't mean they don't have such a thing.

Wouldn't that be renting a shovel vs selling a shovel?

NVIDIA sells subscriptions...

I'm only aware of Nvidia AI Enterprise and that isn't required to run the GPU.

I think it's aimed at medium to large corporations.

Massive corporations such as Meta and OpenAI would build their own cloud and not rely on this.

The GPU really is a shovel, and can be used without any subscription.

Don't get me wrong, I want there to be competition with Nvidia, I want more access for open source and small players to run and train AI on competitive hardware at our own sites.

But no one is competing, no one has any idea what they're doing. Nvidia has no competition whatsoever, no one is even close.

This lets Nvidia get away with adding more vram onto an AI specific GPU and increase the price by 10x.

This lets Nvidia remove NVLink from current gen consumer cards like the 4090.

This lets Nvidia use their driver licence to prevent cloud platforms from offering consumers cards as a choice in datacenters.

If Nvidia had a shred of competition things would be much better.

I'm not sure why you're getting downvoted. It's very clear that Nvidia is moving towards directly offering cloud services[0].

[0] - https://www.nvidia.com/en-us/data-center/dgx-cloud/

> If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone.

While I do not actually think Google's chips are better or close to being better, I don't think this actually holds?

If the upside of <better chip> is effectively unbounded, it would outweigh the short term benefit of selling them to others, I would think. At least for a company like Google.

> why wouldn't Google be selling shovels

They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs.

> I can see Google's custom chips are 15x to 30x slower to train AI

TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for inference to serve their billions of customers is far greater than training costs for models - even LLMs. They've been running model inference as a part of production traffic for years.

Google most certainly uses TPUs for training.

This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling

Multiple types of TPUs.

(I work for Google, but the above is public information.)

Because people generally don’t use TPUs outside of Google. The tooling is different, the access is metered through GCP, etc.

Nvidia is in a vaguely unique position in that their products have great tooling support and few companies sell silicon at their scale.

Correct, I'm pointing out politely that's in conflict with the person I'm replying to.

Possibly naive but I very much view CUDA and its integration into ML frameworks being nvidias moat

That’s not necessarily true. Many companies make chips they won’t sell to support lucrative, proprietary offerings. Mainframe processors are the classic example.

In AI, Google (TPU) and Intel (Gaudi) each have chips they push in cloud offerings. The cloud offerings have cross selling opportunities. That by itself would be a reason to keep it internal at their scale. It might also be easier to support one, or a small set, of deployments that are internal vs the variety that external customers would use.

Intel, AMD and others also have chips for training that perform close to or sometimes better than Nvidia's. These are already in the market. Two problems: the CUDA moat, and, "noone gets fired for buying green".

Is that why apple sell there chips to everyone??

Reply on news.ycombinator.com