Nvidia Nemotron 3.5 Lightning and NeMo Switchyard
blogs.nvidia.com
blogs.nvidia.com
I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though!
Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like.
(Frontier models in comparison will write the app in one shot with no errors at all.)
Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs. dense makes a huge difference in how they actually perform. IMO they should use a suffix, like Qwen 3.6-moe vs. Qwen 3.6-dense, or maybe Qwen 3.6-fast vs. Qwen 3.6-smart...
[0] EDIT: Turns out Laguna XS is MoE, I misunderstood. It performed similarly to the dense models. But maybe this explains why it couldn't write code and think about monkey shapes at the same time!
I think I missed that and assumed it wasn't because it performed similarly to the dense models. Interesting!
Also try fine-tuning small models and see how they perform.
Don't use ollama. The entire project is just a series of stupid decisions like this.
vLLM has the best performance if you can fit your entire model into VRAM. llama.cpp is the usual go-to if you're partially loading into RAM. LM Studio is a sensible front-end to llama.cpp. ik_llama.cpp has more advanced CPU quantization strategies than llama.cpp. If you're running super large models mostly from RAM, ktransformers can sometimes be the highest performer, if it works for your model.
Both MoEs and dense models are always getting better, so I don't think this comparison is meaningful across generations. But still for a first approximation, this tends to hold (you wouldn't expect a lot from a 10B model in coding yet).
I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
They weren’t quantitatively better than previous companies. They were qualitatively better.
I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".
And I'd argue "hardware advances" are more proof of optimization.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
I got the feeling laptops gonna feel very different in 2036.
I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.
With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.
I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.
LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
Maybe we'll have model weights being shared like game cartridges.
This has a lot to do with how MCTS works BTW. The current best path is often only the current best path because a lot of investment has been sunk into it. If you were to put equal resources into a different path you may find that it was actually far better. It is just that the early rollouts favored the other 'best path' so you sunk a lot of resources into that one. We are very early in our exploration of LLM architecture. I highly doubt we are anywhere near the best path right now.
The diffusion models are interesting, but those also seem hacky.
I think the next form of AIs will be simpler and more abstract.
The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.
I think first step is to find a better way to represent information.
LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.
The other option is maybe hook up humans to EEG or the likes and map their brains while they solve different kinds of problem, or just see and feel the world around them
Well, everything—even human life on Earth—is just temporary, but RAM supply lagging centralized-AI-driven demand increases continuing to squeeze the consumer market may not be a short term phenomenon.
> LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
So, the solution to the RAM crunch is “everyone has their own home chip fab and deals with the raw material supply and hazardous waste disposal”?
I...don't imagine so.
My idea was more like you get some pre-made chips, that you can maybe assemble together configure at home with your desired models.
Maybe each one of this chip is a layer, so you can stack as many layers as you want.
If, hypothetically, the thing you wish existed, why wouldn't it be valuable to high-utilization cloud providers?
And if useful to them, why would it be sold at a lower price than the (extremely high) price those cloud providers can afford to profitably pay?
The only solution to the hardware crunch that eases the cost/access barrier is if there's such a glut of excess supply of newly-commoditized inference hardware that the price craters.
And you've yet to propose a feasible manufacturing scenario through which that would be possible.
I hope the supply will increase at some point, I doubt cloud providers will be able to buy everything, especially if the competition will be high and the price of AI goes down, so there won't be an easy way to make profits by serving models.
That's the only way I see things going, which ultimately comes down to the market losing faith in AI as a profit engine.
Which is probably going to take one of (a) OpenAI / Anthropic IPO failing or newly public financials looking bleak or (b) a high-enough profile large AI company failing (unfortunately most of the large enough to shock the market ones have diversified and durable cash generation engines).
If we could just do the first part and get reality to conform, the rest would be superfluous, but I don't think we can do that.
There's no immediate fix for the RAM supply issue, but as long as it's manufacturing issue, not a raw materials issue, time always solves it.
This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.
Many on HN are in deep denial about this imo.
Smaller models are suitable for simple tasks like classification / summarisation while larger models are better in agentic capabilities.
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
prompt-cache won't work with these
You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.
You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.
Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.
Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.
First principles of model routing: https://try.works/first-principles-of-model-routing
role-model router and protocol: https://github.com/try-works/role-model
note: edited to keep the answer to the below message clearer
Source?
The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.
Given model A with cache C(a) and model B with C(b)
Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?
I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:
* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?
Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.
But yeah I'm skeptical all this overhead is worth it.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.
The nice thing is that it makes sense for specific setups, less conversation oriented.
As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.
You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.
- problem: massive deluge of information because of AI
- solution: human beings should adopt a minimalist style of communicating in writing.
- e.g. this entire website page can be ten bullet points.
We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.
If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.
I'm not sure people really want that
Or do you have something else in mind ?
Communicating an idea concisely is difficult. Most people struggle to get ideas across at all, asking them to do it well with fewer words is often out of reach.
https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
The cost is similar vram footprint I guess (?)
As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
I think the tps difference between them (both fitting in vram) won't be more than 2x in practice.
I would happily take 20tps over 40tps, if the model gets 3x more correct answers.
You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention).
What was the result with MTP?
Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.
One somewhat related thing is that, without drafters (the models doing just generation) ts tends to slowly go down, while with drafters, as the model is "checking drafts" instead of pure generation, even when the avg ts has fallen to say 50 ts (27B, ~128k ctc for example), there still tend to be random spikes to up to 70 (mostly during coding), which is always funny to me but also makes models a bit more "resilient" to the slow degradation of ts.
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
Happy to hear what would make the website more useful.
Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful.
I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?
but
Qwen 3.8 27B is dropping this week...
Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model.
Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.
I will add model sizes (total/active params) for each model, good point.
If the entire model fits in vram, won't the tps be comparable?
In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.
Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).
Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).