MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference
mkone.ai
mkone.ai
[1] https://www.techpowerup.com/gpu-specs/radeon-instinct-mi210.... [2] https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686
Note that the comparisons between the two cards (MI210 and A6000) are being made for high throughput workloads, and in this regime the performance is compute bound. So as long as the memory bandwidth is decent (as it is for GDDR6 with 768.0 GB/s) the lower memory bandwidth is not the main bottleneck. There are also other architectural differences so any comparison will ultimately be imperfect, but we found A6000 to be the closest match for the workloads that we cared about (ie high throughput workloads at reasonable latencies).
Also worth noting that there are still more stack optimizations on the table, which again can shift the bottleneck between compute and memory. In those cases it might makes sense to compare with another card with matched memory bandwidth.
paul @ mk1
https://aws.amazon.com/marketplace/seller-profile?id=seller-...
For AMD, you'll have to wait until these cards become available on the cloud.
Although that is mostly wishful thinking. I expect ROCm probably works quite well on the server, the issues in my experience is that it assumes that the AMD card is dedicated to processing and once it tries to run graphics and compute at the same time it becomes Too Hard to manage memory for both and then the driver locks up. So their buggy drivers basically kill any sort of hobbyist or researcher effort unless there is a dedicated supercomputer on hand. Much better plan to go with Nvidia I assume.
Simple stuff that matters more for most folks outside FOSS circles.
It is up to Intel and AMD to provide similar experiences.
That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia hardware they reference for comparison. Compare to real (datacenter) Nvidia GPUs that aren't three years old and prepare to get your hair blown back.
I don't have an A6000 but as an example with the tensorrt_llm backend for Nvidia Triton Inference Server (also free and open source) I get roughly 30 req/s with Mistral 7B on my RTX 4090 with significantly lower latency and I'm in the early stages of tuning. Comparison benchmarks are tough, especially when published benchmarks like these are fairly scant on the real details.
TensorRT-LLM has only been public for a few months and if you peruse the docs, PRs, etc you'll see they have many more optimizations in the works.
In typical Nvidia fashion TensorRT-LLM runs on any Nvidia GPU (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM. It even works on their Jetson line of hardware.
You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic.
Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back and then some as a dividend via the software especially when anything close (assuming this even is) is another paid product/SaaS/whatever their monetization strategy is.
At the risk of this turning into an Nvidia sales pitch Triton will do the same thing for absolutely any model via the ONNX, TensorRT, Pytorch, Tensorflow, OpenVINO, etc backends.
I have an implementation generating embeddings via bge-large-v1.5 that's also the fastest thing out there. Same for Whisper, vision models, whatever you want.
I feel like MK1 must be aware of TensorRT-LLM/Triton but of course those comparison benchmarks won't help sell their startup.
[0] - https://github.com/NVIDIA/TensorRT-LLM
[1] - https://github.com/triton-inference-server/tensorrtllm_backe...
[2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read...
[3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...
Apple and Microsoft's inability to work together was exploited to comical extremes, and now Nvidia is laughing their way to the bank with a 10+ year lead on software investment. Even if we're training models on Windows and MacOS in 2028, nobody will have actually solved the HPC problem. The "Linux moment" for general-purpose GPU number crunching may never arrive, certainly not without support from major OSes.