Are they so far ahead?
AMD GPUs get comparable results as of late on Stable Diffusion.
Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science.
AMD GPUs get comparable results as of late on Stable Diffusion.
Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science.
I used to think like that, until I got a job there and... Oh, boy! I left five years later still amazed at all the ever more mind bending ways you can multiply two damn matrices. It was the most tedious yet also most intellectually challenging work I've ever done. My coworkers there were also the brightest group of engineers I've ever met.
What does this mean? Tedious is pretty much the opposite of “intellectually challenging” when I think of careers.
I made the mistake of buying an A770 from Intel, based on the spec sheet. Hardware is comparable to what Nvidia is selling, for 70% of the price.
It's basically a useless paperweight. The AI software crashes constantly, and when it's not crashing, it performs at half the level of Nvidia's cards.
Turns out that drivers and software compatibility are a big deal, and Intel is way way behind in that arena.
I hope but do really foresee that AI/ML systems will have similar competitive stack like Nvidia.
The recent Stable Diffusion results are great news, but also don't include comparisons to an Nvidia card using the same optimizations. Nvidia claims that Microsoft Olive doubles performance on their cards too, so it might be a bit of a wash: https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvi...
Plus, none of those optimizations were any more open than CUDA (since it used DirectML).
> crunching 4/8/16 bit width numbers is no rocket science.
Of course not. That's why everyone did it: https://onnxruntime.ai/docs/execution-providers
The problem with that "15 competing standards" XKCD is that normally one big proprietary standard wins. Nvidia has the history, the stability, the multi-OS and multi-arch support. The industry can definitely overturn it, but they have to work together to obsolete it.
I bought a RX 6800XT to do some AI work because of the 16GB VRAM, and while the VRAM allows me to do stuff that my 6GB RTX 2060 wasn't able to, on performance side it's actually a downgrade in many aspects.
But the main issue is software support. To get acceptable performance you need to use ROCm, which is Linux only. There was some Windows release of ROCm few weeks ago, but I am not sure how usable it is and none of the libraries have picked up on it yet.
Even with a Linux installed, most frameworks still assume CUDA and it's an effort to get them to use ROCm. For some tools all it takes is uninstalling PyTorch or Tensorflow and installing a special ROCm enabled version of those libraries. Sometimes it will be enough, sometimes it wasn't. Sometimes the project uses some auxiliary library like bitsandbytes which doesn't have an official ROCm fork, so you have to use unofficial ones (that you have to compile manually and Makefiles quickly get out of date). Which once again, may work or may not.
I have things set up for stable diffusion and text generation (oobabooga), and things mostly work, but sometimes they still don't. For example I can train stable diffusion embeddings and dreambooth checkpoints, but for some reason it crashes when I attempt to train a LORA. And I don't have enough expertise to debug it myself.
For things like video encoding most tools also assume CUDA will be present so you're stuck with CPU encoding which takes forever. If you're lucky, some tools may have a DirectML backend, which kinda works under Windows for AMD, but it's performance is usually far behind a ROCm implementation.
However AMD is still as terrible at H264 as ever, and their AV1 encoder also has a hardware defect they’ve patched by forcing it to round up to 16 line multiples, so it is incapable of encoding a 1080p video and instead outputs 1082p that won’t be ingested properly when streaming.
http://freedesktop.org/mesa/mesa/-/issues/9185#note_1954937
Also the AV1 quality is not as good as intel+nvidia even with resolutions that aren’t glitched. AMD seemingly went big on HEVC (supposedly because of stadia?) but everything else is a mess.
And most places won’t touch HEVC because of the licensing costs. Microsoft makes it a windows store plugin you have to buy separately etc. somewhat odd that google picked HEVC for stadia but I guess those customers are actually directly paying you vs YouTube being a minus on their balance sheet (at least until recently possibly)