I absolutely love it. I’ve been up for days playing with it. But there are some bleeding edge issues. I tried to write a balanced article. I would highly recommend for people that love to get their hands dirty. Blows away any consumer GPU.
I have H100s to myself, and access to more GPUs than I know what to do with in national clusters.
The Spark is much more fun. And I’m more productive. With two of them, you can debug shallow NCCL/MPI problems before hitting a real cluster. I sincerely love Slurm, but nothing like a personal computer.
As for debugging, that's where you should be allowed to spin up a small testing cluster on-demand. Why can't you do that with your slurm access?
It's remarkable what can now be done on a whisper-quiet little box. I hope the Strix Halo's will be just as much fun, and they should be, so long as Flash Attention works.
Fair, thanks for the answer.
The bane of my existence...
salloc: Granted job allocation 1978
salloc: Waiting for resource configurationEven ignoring GPU details spark is an awesome little quiet powerhouse arm64 workstation that is 100% Linux first
Curious though how you offer idrac to customer, do you have another OOB BMC for the idrac? Or is this internal engineering context
We rent bare metal on-demand and our whole business is to be able to offer compute that you probably wouldn't be able to host in your house $, as if you own it yourself.
So, we made it so that users can get access into the BMC and modify the box however they want. When they are done, we've automated the reset as well. Fully self-service.
$ These boxes are very expensive, weigh 350lbs, sound like a jet engine and consume ~10kW.
That’s a big part of why we hot stage things for customers.
But it’s still not quite like exclusive access to resources when you want them. So I can see it from both ways.
Nah. Do you have 1st hand experience with Strix Halo? At less than 1600€ for a 128GB configuration it manages >45 tokens/s with gpt-oss 120b. Which is faster than DGX Spark at a fraction of the cost.
But please have your LLM post writer be less verbose and repetitive. This is like the stock output from any LLM, where it describes in detail and then summarizes back and forth over multiple useless sections. Please consider a smarter prompt and post-editing…
I was not familiar with the hardware, so I was disappointed there wasn't a picture of the device. Tried to skim the article and it's a mess. Inconsistent formatting and emoji without a single graph to visualize benchmarks.
I bet the input to the LLM would have been more interesting.
It looks like it worked? Why's it say this?
> Verdict: Inference speed scales proportionally with model size.
Author only tried one model size and it's faster than NVIDIA's reported speed at a larger model. Not really a "Verdict".
> Verdict: 4-bit quantization is production-viable.
That's not really something you can conclude from messing around with it and saying you like the outputs.
> GPU Inference is Fundamentally Broken
Probably not? It probably just doesn't work in llama.cpp right now? Takes a while reading this to work out they tried ollama and then later llama.cpp, which I'd guess is basically testing llama.cpp twice. Actually I don't even believe that, I'm sure author ran into errors that might be a pain to figure out, but there's no evidence it's worse than that.
But then it says this is the "root cause":
ARM64 + Blackwell + CUDA 13.0 = Bleeding Edge
↓
Limited production testing
↓
Edge cases in numerical precision (inference)
↓
Memory management issues (training)