Can Google/Tesla/etc create an H100-like GPU in a couple of years at a fraction of the cost? And is CUDA really necessary if this much money is at stake?
[1] https://www.tomshardware.com/news/teslas-dollar300-million-a...
300M hires a lot of engineers. Maybe the fact that they are buying instead of building tells you how hard it is to build and how strong a moat Nvidia has.
Musk has publicly said that the true test of their own training hardware efforts is if the ML team switches to it. I.e. it’s a software, not hardware problem.
> in a couple of years at a fraction of the cost? And is CUDA really necessary if this much money is at stake?
I mean, that's the thing; is resisting CUDA really worth the cost when this much money is at stake?
It's not like you have to pay to license CUDA. The two big drawbacks are that it's proprietary and locks you into their ecosystem; neither of which really matter when shipping stuff at that scale. These companies could spend a few million dollars to accelerate their specific codepath, but it's frankly wasted money unless you have a specific reason to avoid Nvidia.
The eventual Nvidia-killer will probably be platform-agnostic tooling like ONNX, unless hardware manufacturers revive a sort of OpenCL-style acceleration library.
Suffice to say, there are a lot of "competing standards" a-la the XKCD : https://xkcd.com/927/
I don't think any of those will really topple CUDA, though. The best way to unseat Nvidia's dominance would be to target their two weaknesses; the closed nature of CUDA and the lock-in to Nvidia hardware. It would be difficult to overturn them both, but an open and fast CUDA alternative is really all people actually want. That's why I think libraries like ONNX have the right idea; instead of relying on chip manufacturers to not rip each other's throats out (they won't), they unify everyone's proprietary APIs. Barring some ground-up GPGPU library like OpenCL, this seems like the smartest path to me.
We get equivalent performance on non-NVIDIA chips without it and most of the stuff is abstracted away these days
We do write CUDA when needed but really only a handful of folk will actually train models as it is a pain
The real reason for the moat comes down to a number of things, like:
- The flexibility of generic GPU and ML acceleration primitives
- The availability of systems with hundreds of terabytes of GPU memory for you to scale to
- The struggle of trying to use commodity hardware for actual ML acceleration
I deploy models to freely-provisioned ARM servers, I don't think I'm a choosing beggar in the slightest. When I deploy to Nvidia hardware though, the experience is much nicer on-the-whole. This stuff is definitely possible with consumer hardware, ROCm acceleration or CoreML optimization, no doubt. It's not hard to see why Nvidia is at the mountaintop right now though, and unless the industry agrees to stop building CUDA-style moats then this is the history we're damned to repeat.