Off-topic and somewhat of a rant, but I'd far prefer us all focusing on open standards like HIP, SYCL, OpenCL, etc.
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
I would really appreciate your input!
1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think)
2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input.
3. Make sure it has access to reference implementation code
Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.
Can you tell us what kind of gain you got ie. what was the speed bump
Keep in mind though that this is also sometimes use case dependent. Off the shelf implementations are generally pretty good overall, but can have pathological behavior on specific workload shapes you care about. So do this as a somewhat later optimization, and particularly when you see performance characteristics that don’t seem to make any sense.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
* manual pipelining
* shared memory staging
* tiling
* bounds checking
Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture.
See https://www.bit-tech.net/news/gaming/pc/carmack-directx-bett...
The moment LunarG no longer gets sponsored it will be back to "here is a standard, all the best, the community will provide the tools".
On the SYSCL front, that is mostly driven by Intel, and their own OpenAI tooling, there are hardly any offerings for SYCL from anyone else. Thus it is really only pseudo-open.
In that regard, the Vulkan SDK is definitely an improvement.
However using Vulkan on Android is actually hardly any different, given that there is no SDK as such, only the libs and headers get dumped into the NDK, there is a github with samples from Google and that's it.
So bad that they failed to convince anyone to drop OpenGL ES from Java/Kotlin, and use NDK alongside C or C++ for Vulkan, thus now there is WebGPU implementation for Java and Kotlin consumption.
I do agree with the Vulkan code itself, related extension spaghetti, and overall state of drivers, which has become visible enough to have been part of Vulkanised talks for two years in a row now.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
Intel uses SPIRV iirc
The basic underlying tech can be agreed on, but Apple/AMD/Intel all have different GPU priorities that limit their ability to agree on a CUDA-adjacent hardware platform.
Intel (via Codeplay) was handing it out on a silver platter - Nvidia on SYCL, full top chain, and people still wouldn’t want it.
By holistic, I mean hardware architecture cooperation. Nvidia can hold onto their lead forever if GPU designers fight over what a GPGPU hardware baseline looks like. The current ecosystem fragmentation is not competitive, and future fragmentation probably wouldn't work either. I think the fastest way to kill Nvidia would be a hardware consortium.
What would you expect the hardware consortium to coordinate on? Unified ISA?
Yes they have implementations on top of CUDA but they're maintained by... Intel. They didn't get buy-in for cross-vendor collaboration
Intel missed for a different reason - they didn't invest in the product space at all for decades. In terms of units they had the most market share (millions of Intel Integrated Graphics chips), but it wasn't seen as important. Bare minimum to render Windows and Office UI, nothing more.
They're migrating away from SPIR-V to their own, Intel PISA: https://discourse.llvm.org/t/rfc-upstreaming-the-pisa-backen...
Vulkan's SPIR-V dialect is substantially different from the OpenCL one, notably with the former having structured control flow. They're incompatible between each other.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
My thinking was rather that most of our current programming languages put memory layout fully into the programmer’s responsibility - I can think off hand of a language where the compiler makes performance decisions like whether your structure are SoA, AoS or SoAoS, what alignment, padding, strides and float types to use.
Automatic decisions about when to use cooperative loads through shared local mem versus gathers from global mem and hardware caches are also something that such a hypothetical compiler would have to make.
As for ai and matrix vector performance... I mean matrices are extremely useful across many domains. The hands off language that exists today is called blas. That's fine but won't lead to the best perf on any GPU today.
SoA and AoS data layouts are not even a worthwhile point of contention. Same with shared v global mem. Today's hardware has specialized memory depending on the operation. The hardware on these processors is so specialized as to make anything but first class support for the feature be completely pointless. If you look at Nvidia code that's open source even they will special case a lot of their chips. Literally if you want the best perf you write the kernel exactly for the exact chip. That's intra vendor .. you can only imagine inter vendor
Early graphics APIs ran on GPUs that were just about as different form one another as that.
We could easily figure it out, were it not for the trillion dollars incentives to not figure it out.
Sucks like important libraries like Alphafold are locked into CUDA. Its ridiculous for researchers.