Hacker News Reader

Comment by maxignol | Hacker News Reader

Optimizing speed is really the way to go. Yet 24GB is not what everyone can afford. Maybe we could take some of those 56tk/s and transfer into some free RAM space using MoE loading ? I'd be glad with a less than 10GB and more than 6tk/s model.

View on HN

Unfortunately this is just the entry price for LLMs. With the exception of the Qwen 27B models, I personally haven’t found a ton of use cases for models less than 200B. With the right setup, fine tuning, etc, you can make small models do cool things, but hard to please everyone given the insane hardware costs at the moment and the comparably cheap API costs.

Small models are still great for lots of “simple intelligence” use cases, like annotating or summarising files and media; or even just basic chat when given web search tools.

My local NAS is private and I’m not going to send it off to APIs for captioning or metadata; but even Qwen3VL 8B does an excellent job at this, despite being quite old.

They are also really excellent for fine tuning. Unsloth and Tinker (from Mira’s TML) are great places to start.

If your use case is narrower than “coding agent for everything”, you can probably match frontier performances on that narrow domain with ~30b and exceed it with ~100b+.

Gemma4-E4B (4B params) works pretty well as a local wiki, or when you don't have connectivity.

Nitpick: Gemma4-E4B is actually a 8 billion param model, but only 4.5B params worth of memory bandwidth needed per decode.

some small models are fast, and fine tuning can be done locally

for example in gaming context, if you need an answer below 5 seconds, they are the sweet spot

I don't thinnk just MoE will solve it. If you hit constantly different expert layers, you can't outsource layers efficently and have to swap it in.

MoE will be faster because it will read less memory for sure, you still have to have it though.

Reply on news.ycombinator.com