For anyone wondering “how slow is this?”
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
Weeps...
For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.
So what you get is 80x more expensive and you wait 416 days to get it.
Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.
MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.
The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.
Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171
Like there is no explicit goal aside from each 'expert' getting roughly equal weight?
And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?
I do not know what I am talking about, this is my limited understanding...
Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.
Huh, always thought one would sort of require the other.
For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories.
E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session.
There will also be languages that are requested very often and others that are extremely rare.
(Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed)
So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
was this meant to read "sharding"?
If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful.
If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
If you had a deadly condition, and no diagnosis worked, and a specific model had the answer... past that I wouldn't use it.
I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.
I have all praise for those taking this on and in my idiom would call it *the lord's work."
The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.
It's work like this that moves things forward.
But I agree, what the OP does is a lot more efficient than this.
the canary in the coal that this has not change is nvidia share price.
https://raw.githubusercontent.com/timm/ourmine/refs/heads/ma...
that has 93 NASA software projects, you are looking at less than 9 LOC per engineer/day
9 LOC per engineer/day...yes its about 5 to 6 min per token. Do you do better?