Hacker News Reader

Comment by adamtaylor_13 | Hacker News Reader

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you.

Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

View on HN

Local models embody the hacker spirit, constant Claude glazing is spiritually incompatible with tinkering. Don't upload your spirit to the cloud.

But if I run the model locally I have to pay for it, whereas with Claude I can, oh wait I just hit my 5 hour free limit with 2 messages.

They may well do but in practice if you want to embody the hacker spirit, the best thing is to hack rather than trying to get some clearly inadequate local LLM to do it.

That’s like saying cloud computing is spiritually incompatible with tinkering.

It is. You can't tinker with someone else's machine.

You can’t tinker with someone else’s machine but you can tinker through it.

Yo, MTP for Qwen is sick, thank you! Your work is invaluable.

I experiment a lot with local models, and I agree.

I have a lot of fun with the local models and seeing what they can do.

I appreciate the SOTA models even more after my local experiments. The local models are really impressive these days, but the gap to SOTA is huge for complex tasks.

What if you split it into less complex tasks? E.g. use the model to help decompose the task into parts, then help it iterate through it.

Gives you more control over the outcome and more steering anyway.

Reasoning over a large codebase is only one use case for large models. For the use cases in the article (summarizing, classifying, basic text rewrites) most phones can handle them just fine.

DeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There's hope that Opus 4.5 level performance locally is not that far away.

From what I read, ds v4 is very close with opus 4.6 performance.

The full model is, not the quantized versions.

yeah that goes without saying. how can openweight, quantized version beat SOTA :)

Well it depends on the task. For agentic coding, more is more, but for tasks that normal consumers use them for there really is a ceiling. OCR, text to speech, that type of thing doesn't really improve when going to a SOTA model, so you'd just be wasting your money. I think local LLMs have more value than software engineers give them credit for.

totally agree with that. local llm doesn't need to match SOTA performance in order to be useful.

Running DeepSeek V4 without extreme quantization locally requires a lot of hardware.

The IQ2 quants that fit into 128GB machines are very degraded.

That is true, it is a 1.6T parameters model so it requires a great deal of memory. I also heard there's a 2bit quantization that works well on Apple metal.

The article is not about those use cases. There are plenty of use cases for which local models are already pretty good

Should be relatively quickly, 1-2 years for local models to catch up to today's SOTA.

Of course then you'll be asking "uhh lemme know when Opus 6.8 level performance is available locally". People are never happy.

Gemma 4 and Qwen 3.6 are legit beast models that would steamroll every API offering from 2 years ago.

Depending on a task, there are already models matching Opus 4.5. Just not in everything. But you can always swap a local model for a particular task.

Opus is probably somewhere in the 5TB parameter range and needs terabytes of GPU memory.

The economics of running SOTA locally just does not make sense, because you’re not using it 24/7 at 80%+ utilization while the cloud based providers can.

Next year there will be Opus 4.5 level available on open source models so theoretically you may be able to run it locally but in reality it will be too expensive (i.e maybe 2 x max Studio 512GB ram each) for “normal” users.

The frontier Chinese open source models are already at this level, GLM-5.1 and Kimi K2.6 specifically.

But you can't run the locally at full quality. And quantized versions you can run locally are a far cry from Opus 4.6.

Anthropic serves quantized versions of their models and you can run q8 locally.

I don't even use Sonnet anymore. Current feels worse than Claude 3.5 couple years ago. They have quantized that much? Switched to GPT 5.5, let's see how long it will stay good.

The problem with GPT is it doesn't have the same wholistic worldview of the problem space as Claude

Hmm, what that means or how can you even measure it? At least for all my recent problems, GPT 5.5 has performed better.

Reply on news.ycombinator.com