AI chip making? I can train an AI on my intel laptop if I whish. If I need more CPU power, i can rent some.
The genie is out of the bottle and the only way is forward. The latest worldwide race.
This isn't accurate. The bottleneck in very-large-scale-training BY FAR is communication between devices. If you have a million CPUs, the communication cost will be significantly higher than a thousand A100s (perhaps in the order of 100x or even more). So this is only possible to replicate with very dense and high compute chips with extremely fast interconnect.
Thanks for providing this insight. Is A100 the only platform? Can we pause/resume all such platforms simultaneously?