Hacker News Reader

Comment by jacquesm | Hacker News Reader

LLM threads the world over are spammed with Strata links, it remains to be seen how much of the breathless hype remains standing once the honeymoon period is over. I've tried it but so far I have not seen anything that overly impressed me in terms of accuracy, though the speed is definitely there. I'm sure there are applications for LLMs where the quality of the answers is less important but I don't have any of those. YMMV.

View on HN

Qwen models famously chase benchmarks. They are good at very specific tasks, but they aren't good problem solvers. The reason Strata works is because of those MoE models, which basically shoehorn themselves into less intelligent versions.

Meanwhile models like Gemma4 which are fully activated preform slightly worse on the specific benchmark but are capable of exploring much more domain space with careful prompting.

If you have a card with 24gb of ram, you can run Gemma4:31b above 100 tok/sec if its not being used for rendering. With careful prompting and RAG system, you can actually make it do better than frontier models. Especially if you add web search into equation (which with google basically runs Gemini for you).

I'm sick of these 4 day old vibe coded projects becoming hyped to hell while I spend a year on something and it goes nowhere

I don't know who this kid is but I've looked at the code, it's all Claude. Saying you can run on 2-bit quant with some optimization flags at 100t/s isn't like some "oh my goodness" ... it's wasting everyone's time with noise. I mean give me a break.

What are they doing? Is it Tiktok? Discord? LinkedIn?

I want to do high quality work but apparently I should be dicking around on social media

There are quite a few of these, indeed. Most of them either start of with llama.cpp (everybody's favorite to rip off, make a minor improvement to and try to establish a reputation) or vllm. But some of these actually do have something to bring to the table, for instance, by limiting the themselves to be able to focus on a smaller set of possibilities and this can lead to favorable outcomes for a smaller size of the audience. One particularly good example of this I think is 'ninfer' which mainly focuses on the Qwen family of models served up on RTX 5090's, which it does outstandingly well. Other people then go and fork that to adapt to their hardware so now there are 3090 and 4090 forks of ninfer.

Then there is the 'unified memory' branch of inference engines, the most notable of which is probably DwarfStar 4 by 'Antirez', which also started off with a lot of code from the llama.cpp codebase.

And then there is 'the rest', but even there, some of these have interesting bits and I always hope that eventually those bits will make their way back to the engine where it started.

I run both ninfer and llama.cpp, vllm is an unmaintainable mess even though there usually is a performance edge (it is great if you are serving up for commercial purposes so you can tweak it for one set of hardware and one particular model). You'd essentially need to dedicate a week or more to getting a new model up and running on a particular set of hardware if it does not nicely match with the recipes found online.

One exception is the DGX Spark series, there vllm is supported by the manufacturer and the hardware is very consistent from one box to another. But the performance isn't really there when compared to a fat PC with a bunch of GPUs. (The 200 GB/s memory bandwidth is a serious performance killer).

"I'm sick of these 4 day old vibe coded projects becoming hyped to hell while I spend a year on something and it goes nowhere"

You know, in a way, you are both in the same boat. The difference is that the 4 day old project gets the clout first and then becomes irrelevant.

Same could be said about models… every now and again there’s hype around a new model, then you look at the card and it’s a 27B or 35B-A3B without a single mention of Qwen in the text.

I think the problem is maybe being the cock.

What does this inference engine do that others don't?

It is faster than comparable engines using the same model, but they use a lot of short-cuts.

Reply on news.ycombinator.com