Hacker News Reader

Comment by girvo | Hacker News Reader

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

View on HN

I have a M1 Macbook Pro...with only 16gb and I struggled with Qwens2.5-14b trying to do large projects. I loved Qwen but I had to try and do something different. So I switched to Gemma4-12b which looking at it now, seems more like a downgrade than an upgrade.Can you refer me to any Qwen coding models that wont choke my poor 16gb and also connect contextually? I need that context. I love the laser point focus, but I need context and basic understanding of that context.

I don't think "large projects" is realistic with a model that fits in ~8GB (I'm assuming you run stuff other than the model). And, Gemma 4 12B QAT at 4-bits is surely the smartest model in its size, but it shines at vision tasks rather than agentic tasks (though it is a good tool user and can do stuff like research, it's obviously not aimed at code).

You can almost always find free models on OpenRouter. Google AI Studio also has free usage of Gemma 4. Both are rate and usage limited which agentic use will probably chew up pretty quick, but you can usually find some pretty powerful models for free. If you rotate through different providers, I think it avoids the cap. Currently several Nemotron models, North Mini Code, Laguna models, Gemma 4 31b and MoE, Qwen 3 Next, and gpt-oss 120b, are all available free on OpenRouter...and better than anything you can run locally in ~8GB.

WOW. That was alot of models. I will be sure to look into those. What caught my eye was the North Mini Code. How is that at reason and context? I guess I will just go see. Thank you for the insight. I am building and learning at the same time, and I really need a model that wont chew up my poor little M1, but at the same time will understand my direction, my context, and also my explanation of need and then be ...' oh, I know what you mean..' then help vibe with me to create it. Qwen was a code model but hallucinated and was never really good at contextual understanding. Gemma seems to get it but isnt necessarily a coding model.(As you said) And I just cant be bogged down with all these fees to get what I need on my journey. So onward on the model search I suppose. Thank you very much for the time.

North Mini Code is on par with the similar-sized MoE Qwen 3.6 and Gemma 4. It benches a little better than Gemma 4 26b a4b and a little worse than Qwen 3.6 35b a3b for most coding-related and agentic tasks. For my use case, or at least the use case I've tested, it is a little worse than both for surfacing security bugs, but all of the MoE models at this size are pretty bad at finding security bugs (they hallucinate a lot of false positives, which degrades the value of their real bugs dramatically).

Poolside Laguna XS.2 is another in this category (30B-35B MoE, reasonably competitive on coding related benchmarks). Also free on OpenRouter. But, also, it's bigger brother, Laguna M.1 (225B A23B MoE), is also free on OpenRouter last time I checked. Worth a look. https://openrouter.ai/poolside/laguna-m.1:free

The good thing about all the 30-ish MoE models is you can run them on any 32GB GPU, even old ones, at a very comfortable speed. A 24GB GPU can run the 4-bit quantizations if you use a quantized K/V cache. That's why there are so many of them. It's the sweet spot for "good enough to be useful for some coding tasks, small enough to fit on the GPU a lot of people have".

The cheapest not rate-limited options that are actually pretty competitive with the frontiers are from DeepSeek and MiMo. DeepSeek V4 Flash and Pro are extremely cheap, their caching is the best in the industry (and their cached tokens are even cheaper), and Reasonix is an excellent CLI harness that is designed around maximizing cacheability of DeepSeek models, specifically. I used it for an hour last night and spent something like three cents. MiMo has token plans that are a pretty good deal (though confusing...the token plan buys credits, and credits are not a whole token, so you get billions of credits on the token plan for a few bucks, but it chews through it at a rate faster than 1 credit per token). But, DeepSeek V4 Pro is a consistently better model than MiMo.

I appreciate the insight. I might look at these a bit later... because I am swamped with this new build Im putting together. I havent yet found my so-called 'sweet spot' yet for larger tasks. I checked out the gemma4 variant the one or two times just for contextual reasoning and smaller build capabilities, but Ive only been testing single use isolated tool tests with my mcp pointed at it lately to ensure tools I've installed actually work before I incorporate them into the mcp server. No plug or anything, but just so you can get a feel of what I'm doing, I initially set out to learn coding. Then I fell into trying to build my own local model to help teach me coding (and anything else related) because tutorials and all these fancy webinars and such just put me to sleep. Once I discovered all of this AI stuff...? My interest in cybersecurity just skyrocketed. So....no life story or anything) ha... I started to build this mcp, pointed at LM Studio with qwen loaded as my first one. As I began to familiarise more I realised...I need a model that is free, local, will understand context, reasoning, can code, de-bug,vuln discovery- etc. etc. All of this sent me on this whole cybersecurity rabbit hole - as it does with any tech stuff, and now along with my need for a model to teach me coding(Python) I wanted a model to also HELP me build ITSELF basically. Because so far, Ive used good old Claude- Sonnet-5 straight from the mobile app for said task. Dont laugh. And I show the build,and upgrades of the mcp to GEMMA (initially qwen) and get fedback against the Cluade assisted builds. So I apologise for ruining your day, but... thats where Im headed. I need a model that will vibe with me, build with me, reason, comprehend context and also... not chew up my m1 16gb, or token usage. Well, if its free I suppose tokens arent a thing but they do sort of matter still. Thanks again. Your information is much appreciated and invaluable.

Sorry, I also forgot to say, to the rest of your comment: Its actually 16gb M1. And yes the entire operation is, all running at the same time. Kali/UTM with an mcp server pointed at my LMstudio with gemma loaded as the local server. And maybe some spotify to lighten the mood while I work. But I learned not to have multiple things running, REAL QUICK. So ultimately I am running pretty healthy and no sort of crashes or choke points. I ran a 14b QWEN in LM Studio w/ no issue. Any higher and I think I would fall off a cliff somewhere.

It would be great if the Gemma folks would release a code-focused model. Probably won't happen, but it's fun to dream.

The Ornith folks say they're doing that, but haven't released the Gemma-based 31b yet (https://github.com/deepreinforce-ai/Ornith-1). But, also, the Qwen-based 35b MoE Ornith version performs worse than Qwen 3.6 and Qwen AgentWorld on my benchmarks (which are focused on finding security bugs, so not exactly the same as agentic coding, but closely related skills).

That said, the reason they're able to release Ornith branded post-trains of both Gemma and Qwen is because they're open weights under a friendly license. Someone, not just Google, could make a coding focused Gemma post-train. I don't think it's actually much weaker than Qwen 3.6 for coding; Gemma 4 31b outperforms Qwen 3.6 27b by a wide margin on security bug hunting (at least for the specific bugs in my benchmarks, which are mostly relatively difficult bugs from the Mythos-reported bugs).

I'd really love to see a bigger MoE from Google, though. A 70b or 120b MoE would likely be super fun.

I know this wasn't meant as a response to me, but I thought I would chime in, if thats okay. Sorry its a bit of time since the timing of this message.And I hope Im not intrusive. But to give you a bit of an update, I broke down and subbed to Claude's PRO feature. I know it defeats the purpose of local and private, but I still have my Gemma in my private LM studio. WOW. The coding diff is like night and day. I had Claude Code audit my repo page and it was quick and production grade. Like it systematically just chiseld away at all the sloppy choppy and dead cade and even polished up the aesthetics of my page presentation. I need this as a local private model.

Ya, doesn't seem to be google's focus at all, right?

gemma is also worse for tool calling. not just coding

That is because they use a different tool calling format than most other models. Unsloth quants fix this in their Gemma releases.

Can you say a bit more about this? The bad tool calling has made me give up on using Gemma for my Hermes and a personal recipe site. I have only downloaded from Ollama.

Ollama is not recommended [0], use llama.cpp or more specifically Unsloth Studio which wraps llama.cpp and which has an API mode you can use to hook into Hermes or another agent. Unsloth make both the Studio and the quants which fix various issues with many models [1] as well as implementing new features like MTP and QAT support much sooner than other teams. In general you should read r/LocalLLaMa as it has a lot of updates regarding local models as the field moves fast.

[0] https://sleepingrobots.com/dreams/stop-using-ollama/

[1] https://github.com/unslothai/unsloth/discussions/4921

Everytime I try to mention how shit Ollama is, I get mass downvoted here by folks that don't want to hear the truth: There are 4 good inference engines (okay, 5, but we don't count huggingface because it's slow):

1. vLLM

2. sglang

3. (nvidia only) TRT-LLM

4. llama.cpp (mac only, the above are better for non-mac)

If you're not using one of the above, you're doing it wrong

I've never been able to fix the tool calling issues. Running unsloth versions with llama.cpp, constant issues. Have tried many forum fixes, including lots of fixed chat templates, to no avail. It's mostly the edit call that breaks, which often results in "let me just rewrite the whole file from context".

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision.

I do like Gemma for translation, however.

Gemma 12B? It's unique in the Gemma family, and unique among vision models. It's a novel encoder-less model...the whole model is vision. Somehow. I don't understand it, but it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close. And, is also tiny and fast, compared to those larger models, so it's better and faster and smaller. Weird combo.

Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some species in trying to identify a plant, and the other guesses it made weren't all that close, while Qwen 3.5 122B-A10B got it right on the first try.

12B did get one right that 31B got wrong. I'd have to do a much more thorough eval to really compare, just a few anecdotal observations and it's kind of hard to really distinguish, but from the samples I've seen, Qwen 3.5 122B-A10B is doing much better at this task.

The 12B architecture definitely is interesting, and it may punch above its weight due to this (though again, would really need to do proper evals to compare). But of the models I've tried, Qwen3.5 122B-A10B really seems like the best for this kind of task.

Ah, yeah, a model ten times as large on disk will know more, for sure. My tasks are more about "what's happening in this image?" rather than "what is this thing?", which doesn't require encyclopedic knowledge, it need good reasoning about what it's seeing.

Reply on news.ycombinator.com