Hacker News Reader

Comment by giancarlostoro | Hacker News Reader

My understanding from the more hardware savvy friends I have is that AMD is better for inference (so when you run a model) vs for training Nvidia is still king. Would be interesting if OpenAI does in fact take AMD's offer and how they use it (if they even share openly).

View on HN

Today, yes, it's true that AMD hardware can be competitive mainly for inference.

The story here is about the future. Over time, OpenAI would benefit if AMD hardware becomes competitive for training too.

Nvidia currently gets to charge whatever the market can bear on its dominant training hardware.

If AMD hardware becomes a real alternative for training, Nvidia will be forced to compete on price.

The reason I think its worth noting is that Inference is definitely what I imagine an insane amount of their computer is / will be spent on. I could see a very optimal setup where training is all Nvidia, but inference becomes reliant on AMD.

Larger memory, weaker comms. You can optimize for this by doing things like increasing batch size/data parallelism vs sharding schemes with more comms.

At scale training won’t be able to avoid comms entirely, while many models can fit in a single MI300 for serving.

> OpenAI

> "If they even share openly"

That fact that this even needs to be pointed out but is normalized in the AI industry. Sadly, in the history of Open AI, they've been anything but open.

For inference there are many solutions like groq. The margins there will be small.

The fat margins is in training, in NVidia hardware.

Inference margins will be small(er) but it will be much bigger in market size than training.

That said, I'd much rather be leading in training than inference, of course. Nvidia still leads in inference, by the way.

Reply on news.ycombinator.com