Hacker News Reader

Comment by CharlesW | Hacker News Reader

I could've sworn that it was once a given that people doing serious AI work needed CUDA-capable GPUs. Is that no longer the case?

View on HN

The article highlights that onnx, deepspeed, pytorch and tensorflow all support these AMD processors

Thanks, so I think you're saying that there are two related things happening that are eliminating NVIDIA's CUDA moat: (1) Developers are mostly using libraries instead of writing "to the metal" (CUDA), and (2) many popular libraries have added support for AMD GPUs. Does that capture it?

It's the opposite in my experience.

Pure PyTorch mostly works OK, but some libraries implementing crazy optimized, hand written kernels and such will have some trouble.

So (for instance) maybe you can run an LLM in a particular PyTorch framework, but flash attention 2 doesn't support your AMD card, so performance and memory use takes a hit.

Or maybe the library works on an Intel XPU with like 5 changed lines in the entire library (rename "cuda" to "xpu"), but no one bothered to add it, or maybe the dev doesn't even want to support the PR.

Not that I maintain any hugely popular deep learning code, but my own "hobby"/dev hesitance at adding multiple devices is that I don't own them to test them.

If I have an NVIDIA GPU, the only PyTorch backend I can promise even "somewhat works on my machine" is CUDA.

AMD also have HIP which is almost exactly CUDA, but target agnostic, so basically now CUDA code will now run on AMD hardware. So they have the low level via HIP, and have also integrated into the libraries like pytorch. It's a great bit chip out of that moat.

https://www.databricks.com/blog/training-llms-scale-amd-mi25...

https://embeddedllm.com/blog/vllm_rocm/

AMD is "CUDA compatible" in theory.

Intel is taking a slightly different approach, and is going for "PyTorch compatible."

You will hear endless negative anecdotes about ROCm/OpenVINO, but they both do seem to be getting better with each update.

Now that AI made Nvidia a trillion dollar company AMD has finally woken up and realized what they should have ten years ago: that they need to invest in the software side of things more. There is movement now, and you can actually do some AI things on AMD hardware, but it will take a long time for them to catch up to Nvidia.

You seem to have forgotten where AMD was 10 years ago. Company was circling the drain but they should have invested heavily in the software stack for hardware they could barely afford to design? Brilliant strategy. Now's the time to invest because they finally have the resources. The ai train is not going anywhere anytime soon

It would have paid for itself a hundred times over in short order. And if they had started ten years ago they wouldn't have been far behind and the required investment would have initially been much less.

Ten years ago they were valued around $1 a share and teetering on bankruptcy. If they didn't ship Ryzen there would be no AMD today at least not what it was or is.

There are a lot of moving parts around all of this. AMD was still dealing with their fab breakup and fallout.

They basically cut all the fat and some muscle to keep pushing forward so I agree with the other poster that now is the time to focus on software and growth.

If AMD 10 years ago tried to compete with NVIDIA and Intel both simultaneously, would it have survived? They did well to stay in business, and now they can address the AI thing.

AMD beat Intel in the server CPU market.

AMD and Apple both have TSMC to thank for their ability to make chips that beat Intel's. Unfortunately for AMD, Nvidia uses TSMC too, so they'll have to invest in software to compete.

They are investing in the software.

https://fortune.com/europe/2023/10/04/amd-lisa-su-nvidia-roc...

That's a bold assertion with no evidence to back it up

I've been running large scale data centers for years now with tens of thousands of machines. I'd personally never put an Intel cpu into something that I cared about. We used $30 cheapo intel cpus for our mining rigs cause they literally didn't need anything more than that. But, anything else we buy for servers, is always going to be AMD. One caveat to that would be if we have some customer that demanded Intel and we'd lose the business otherwise.

So yea, in my eyes, AMD beat Intel. I don't think I need any more evidence than that. ¯\_(ツ)_/¯

There's a big difference between AMD beating Intel in your eyes vs the market.

¯\_(ツ)_/¯

challenge accepted ;)

(hello!)

There aren't enough CUDA-capable GPUs to buy. So people will enter learn how to use products like the MI300X or they won't have AI.

It's also a matter of price. Nvidia sells their top GPUs at insane premiums. For Microsoft which is such a huge player and the major AI vendor and owns a lot of OpenAI they have no point into cornering themselves into Nvidia dependence.

They benefit from competition, not from bending to one vendor.

> Nvidia sells their top GPUs at insane premiums.

It is all sold out, for years. You can't sell something you don't have.

https://www.theregister.com/2023/09/08/tsmc_ai_chip_crunch/

Necessity is the mother of invention.

ROCm support for a lot of AI processes fixed that

That's the joke. It still is. ROCm is nowhere near production-ready and if MS thinks devs will want to waste their time on random errors then good luck with that business. AMD cards are also super expensive so not sure what their competitiveness is supposed to be?

Expensive is not entirely relevant. They are available (for now).

https://twitter.com/sama/status/1724626002595471740

ROCm has also made a lot of advances in recent times.

https://www.databricks.com/blog/training-llms-scale-amd-mi25...

Thread about that tweet (or should I say xeet) from SamA: https://news.ycombinator.com/item?id=38274427

The MI300 is a massive GPU compared to even the new H200.

It was designed as a combined CPU/GPU for supercomputers, with shared memory. But then the AI craze hit, so AMD spun a variant into a pure GPU AI accelerator real quick, which they could actually pull off because the GPU silicon is modular.

...So thats why it cost a fortune. Its really a jury rigged HPC product.

The H100 is, to some degree, also a jury rigged HPC product. The H200 is starting to reach a more "AI" balance between compute and memory bandwidth.

I can see MI300 being used for CFD, nuclear physics simulation, cloud GPU virtualization etc. but for AI it's simply not there regardless of AMD's and its affiliated startups' marketing.

You're thinking the MI250. MI300x is a beast for AI.

Vega was an ML monster on paper as well, handily beating 1080Ti. Then the reality set in...

These aren’t consumer cards. Apples and oranges.

Microsoft can develop their own software. We are talking a company that places billions in orders to Nvidia, they can afford to develop the tools and obviously AMD will throw millions and millions in support to court such a customer.

Reply on news.ycombinator.com