Training LLMs with AMD MI250 GPUs and MosaicML

"Performance was competitive with our existing A100 systems. We profiled training throughput of MPT models from 1B to 13B parameters and found that the per-GPU throughput of MI250 was within 80% of the A100-40GB and within 73% of the A100-80GB. We expect this gap will close as AMD software improves. "

> The MI250 has a larger amount of HBM memory (128GB) than even the largest A100 (80GB). This means the MI250 can hold larger models for training or inference.

AMD put two individual devices together, so you can tell people the combined memory is more than Nvidia has to offer.

> Overall we find that AMD MI250 achieves an average of ~80% of the per-GPU training throughput of A100-40GB and ~73% of A100-80GB

So their two-device bundle with more memory and theoretical compute just can't compete with Nvidia's prior gen single device.

Can MI300 compete with H100?