432 GB of RAM per chip. Tens of terabytes per rack. Clients queuing up to buy them.
RAM prices are not coming down any time soon.
RAM prices are not coming down any time soon.
ZLUDA basically crash on high memory demand, and only accounted for ~70% of CUDA API coverage (and it is still buggy).
But hey, at least it does run on Rust-CUDA. I'm one of the few who ported it and fixed a few bugs on ZLUDA. I used it to run a simple SHA256 kernel and it ran sure, but I gave it up because of those fundamental problems on AMD GPUs. You can't believe how messy ROCm is. I wonder if Vulkan compute kernel using SPIR-V would be a better choice.
> I think the thing that we were thinking about originally was whenever we're bringing up a new hardware platform, it's a big effort. It's like a huge thing. And so as we were thinking about this, we started doing our own evaluation of MI 355. You guys generously got us a rack to start working. And we expected this to be kind of a big process.
> Our actual experience was we had one engineer who start doing it. They spun up Claude, asked it, hey, bring up this machine, left it going over the weekend. And we ended up with a graph of the actual performance of our leading model on it, just going up and up and up over the weekend.
But anything other than that, nah.
Edit: Maybe I remember what algorithm that is. It is a trick to do coding theory stuff by using NTT (Number Theoretic Transform, the friend of DFT, the Discrete Fourier Transform, but on a ring and hence applicable to finite field) on GPU to do computation over a GF(2^8) field, for my former employment but I went back to using GFNI. Still I have a NDA to adhere to and can't really explain what happens.
Variance in numerical instability across hardware is basically a given, because all the different accelerators implement their "fast paths" in slightly different ways. Modern AI just takes it on the chin.
Anthropic, for example, already has to support both Nvidia CUDA and Google/Broadcom TPUs - AMD aside. And I expect those two to have more architectural and software differences than AMD ROCm and Nvidia CUDA accelerators would.
And yes. If you're wondering how they solved the problem of variance between those two: they didn't! Claude instances that run in the TPU land produce slightly different outputs than those in Nvidia racks! Just not different enough for, you know. Anyone to give a shit. No statistically significant effect on model performance.
Just pointing out, there is a reason the large LLMs keep working to try to make it so their own tools can't attack their own moats, by preventing the exact behavior that you dismiss above.
Read the "Chipping Away at the CUDA Moat: Using Agents to Enable Day 0 Support" section of [1] where they list a number of agent-created PRs that have been integrated into production vLLM/SGLang code and as well as AMD kernel fixes.
[1] https://newsletter.semianalysis.com/i/208362548/chipping-awa...
Anthropic has Mythos. That thing's low level code "AI slop" is better than the "meatbag slop" most software developers write, and it can keep cracking at a given problem with persistence.
OpenAI has GPT-5.6, and also that rabid dog of an AI model that was last seen out in the wild tearing HuggingFace open.
Modern LLMs are very, very capable - not just of writing raw code, but also of persistent, methodical problem solving. Which is what you want to tackle things like "port from an exotic system A to an exotic system B and smoke test the port". Persistently hunting for testable optimizations is a good fit too.
That kicked off my PTSD to setup ROCm on my 6600XT and 9070XT, not even with different containers. I have to build an image for EACH architecture. That's duping 15GB for each arch on my already terribly small SSD.
> The shift SPIR-V enables is from ahead-of-time compilation per target to compile once, just-in-time specialize on device.
It still requires a ton more ops time, not every dc handles it cooling-wise, and you have to be careful with the model selection, but when it fits, it fits well