Hacker News Reader

Comment by KronisLV | Hacker News Reader

I feel like most hardware to run LLMs on is shaped wrong for individuals.

It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards would be kinda useful (albeit NVLink or equivalent would need to be commonplace).

Obviously nobody is running Kimi K3 locally without an insanely beefy homelab and lots of money to burn, but running GLM 5.2 would be cool at like ~100 tokens per second for a single session and maybe ~60 tokens per second with N subagents.

How unfortunate.

View on HN

I have found that the "mostly didn't lose anything" Q8 large models that I want to run are all too large to run on the "only $3995!" 128GB max RAM systems that some people are buying, and definitely won't fit with any usable amount of context. Things like Qwen 3.5 122B Q8 or deepseek v4 flash Q8, or Laguna S 2.1 Q8 need 170-190GB of RAM including full context, which fits on a 256GB RAM dual socket workstation or rackmount server (sans GPU).

Copy and paste below from my notes and reported memory consumption with latest llama-server, assuming use of "--no-mmap" to load the entire thing into RAM at the time that llama-server launches.

DeepSeek-V4-Flash-UD-Q4_K_XL via unsloth 145GB on disk GGUF 0.03.323.204 I common_params_fit_impl: projected to use 178175 MiB of host memory

DeepSeek-V4-Flash-UD-Q8_K_XL via unsloth 151GB on disk GGUF 0.02.215.885 I common_params_fit_impl: projected to use 184636 MiB of host memory

Laguna-S-2.1-UD-Q8_K_X via unsloth 120GB on disk 0.01.616.119 I common_params_fit_impl: projected to use 172860 MiB of host memory

Qwen3.5-122B-A10B-UD-Q8_K_XL via unsloth 160GB on disk GGUF 165GB RAM use on launch, fresh context 0.04.976.905 I common_params_fit_impl: projected to use 170038 MiB of host memory

There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1.

Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it gets too much work (at least for me and the time I have available).

> emerging practice of using Q4 quants and Q8 KV cache for local inference

That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.

Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).

IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.

Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?

This tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes.

But it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.

Are you sure the difference is from MoE and not that 3.6 is newer?

Qwen 3.5 27B also scores higher than 3.5 122BA10B. So even in the same generation the smaller dense model outperformed the larger MOE

https://artificialanalysis.ai/evaluations/artificial-analysi...

Ah gotcha, I hadn't noticed that. Thanks

DeepSeek-V4 should use only 5GB for context due to CSA and HCA, see figure here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

But not every framework implements it properly yet.

Yeah, or close enough to 5GB for estimation purposes, for example a just spawned qwen 3.5 122B llama-server instance reports as:

0.07.015.888 I common_memory_breakdown_print: | - Host | 170038 = 162913 + 6740 + 384 |

The 6740 is the cache size.

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting idle most of the time. Don't get me wrong, most desktop workloads are bursty, but the power needed to to them has gotten cheap enough that we can have way overkill for idle scenarios hardware just sitting on our desks.

Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K3 itself.

it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.

Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).

If your workload fits long batches throughout the night you could schedule them better, yeah. But I think very few have a usage pattern like that?

Given the hardware shortage in the world, I suspect renting ("sharing") via APIs will likely remain cheaper for the foreseeable future since each piece of hardware isn't sitting idle nearly as much.

I find it difficult to always have one or more long-horizon tasks 'queued up' and ready to run... I find myself usually bottlenecked on design, review, or something similar that requires me being in the driver's seat. It's possible I could queue up a bunch of tasks, letting the LLM run off in multiple directions, but then I'd be less able to steer and course correct.

Just my experience though, I'm still figuring things out. Perhaps some subsets of tasks would be more ideal for these long-horizon workloads - exploration, multiple competing implementations, etc...

In principle, a slower inference ought to be easier to steer and course-correct. You'd always be able to look at partial results, especially with a local model that doesn't hide its thinking.

Use your favourite harness to help you find long-horizon tasks to have queued up. It's changed the structure of how my projects work a bit, and do you have to do some homework fast of how slow/fast your various providers or local inference are, but it's worth it. Start off with hobby projects so you get a feel for how it works.

I left something gargantuan running over the weekend (decompiling 1980s-era system software) and look forward to checking it out later today when I have a few free minutes.

For software, not only that, but run batches with a cluster of agents working different parts of the same task. Software like Yegge's Gas Town has one agent act as "mayor," and others work on writing or testing various pieces, with all the agents messaging each other. In his book Yegge writes about using up to thirty agents at once.

A decentralized inference network would be cool. Something that's set up so that I can run a model for personal use on beefy hardware, but also farm out the unused GPU time to the network, probably at much lower prices than normal providers since it would be slower and would lack data security guarantees.

Not quite what you are asking for. Instead of selling the unused GPU, you're donating it to the common good.

https://cocore.dev/

I've been feeling for a while that as we keep increasing model size, we're going through the opposite of the PC revolution.

The "democratisation" talk from the frontier labs is especially egregious when they only release closed models (gpt-oss hardly counts) and are trying everything they can to make it harder to run open models.

I agree but worth noting that it's never gonna be very practical to run LLMs like this at home. Unless we have some sort of design breakthrough, the only "sensible" way to run them is at high batch levels on shared HW.

Like, yeah if I could spend a few grand on such a GPU I probably would coz I'm a rich nerd, but I'd acknowledge it as an extremely inefficient luxury, kinda like a sports car.

So I think you could say the real misfortune is that we don't really have the technology (be it computer tech or political/social tech) to do that shared-HW thing in way we can truly trust.

‘Never’ is a big word in the computing world. 10 years from now a model this size will probably run on a high-end phone.

Of course, by then we’ll want to run something commensurately larger.

We could make LLM inference 100x cheaper to run at home efficiently, but that solution might need to be updated every 1-2 years, whereas current GPU are useful for various others tasks and last longer

> We could make LLM inference 100x cheaper to run at home efficiently

do you genuinely think that's going to happen?

"Never" is a long time. Just think about how much ram we had 10 or 20 years ago. 1.5TB isn't a lot really.

The typical ram has surprisingly not increased very much in 10 years.

> April 2016, 8 GB was standard across the 13-inch MacBook Air range

... Now it's 16.

Rich nerds will have quite a bit more. But I suspect the standard of model rich nerds want to use will have gone up somewhat too.

It doesn't matter if you have the RAM, running a 1.5TB model for a single context stream is fundamentally inefficient.

Lots of things are inefficient in IT, yet we do them anyway. Think about the amount of time your laptop is idle.

Post-crypto, the GPU manufacturers took the proactive move to use VRAM to segment the market for the purpose of price discrimination. Sure, data centers will pay vastly more for GPUs, but Nvidia knows that the PC market is steady and reliable. They could get the best of both worlds by kneecapping their consumer cards to tiny amounts of RAM, to dissuade the cloud providers from scooping up all the consumer cards, and then charging the two segments wildly different amounts for what amounts to the same hardware (back when the cost of RAM was negligible)

Yeah, I've been experimenting with DiffusionGemma which sadly isn't as smart as Gemma itself, but holy hell is it FAST, and has image input as well, so doing things like "take a screenshot once per second + ask the model to categorize/model it WITH reasoning before" becomes realistic and doable.

I ended up implementing DiffusionGemma myself with Candle in Rust + CUDA, and it's quite literally the fastest model I've managed to run on my hardware.

The next mac Ultra will allow to run a big model locally with acceptable speed. But we need people to optimize it for that computer, and we’ll be more limited in models we can choose from

128GB is enough to run a large model, quantized, REAPed, with MoE and fast SSD for model weights

Not Kimi K3 large though

128 isn't enough, but if this report is accurate, the next Mac Studio could run it

https://www.macrumors.com/2026/06/25/m5-ultra-mac-studio-202...

They'll just have you buy two studios and thunderbolt them together.

> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session

Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.

An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.

Which quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.

That's about right. Half the TPS with half the power.

GLM-5.2 will be much more demanding tho

If you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it

I have issues with the vulkan backend. It seems to cause over all GUI sluggishness, but maybe that was actually Firefox. I'll give it another go.

You can run GLM 5.2 at a reasonable quant on a cluster of DGX Sparks. On the link below, the guy is getting decent performance and links to a repo. An 8x Spark cluster can run it even better. A Spark cluster is about as good as it gets for (a) runnable at home, (b) "affordable" hardware, (c) not going to kill your power bill.

https://www.youtube.com/watch?v=nbHOBvLlypY https://www.youtube.com/watch?v=PV89U-PNUNA

You're right, and it's interesting to consider why. It's probably a combination of a few factors:

1) Local LLMs are a relatively new phenomenon and hardware takes years. Apple probably lucked into their unified memory architecture being suitable (in terms of memory size and bandwidth) for local LLMs, but it's only with the newest generations we're hearing about LLMs even being a consideration in their design process.

2) NVidia seem to be deliberately blocking consumers from taking this path - as evidenced by the removal of NVLink from the 30x0 series onwards - probably to protect their data center cards from internal competition?

3) Perhaps there's just not the market for it? It's feasible that the number of nerds interested local LLMs is very small in numbers, sales, and profit potential compared to gamers on the one side, and data centers on the other. (This would explain why AMD and Intel aren't trying to out-innovate NVidia in this area, despite it being an obvious opportunity.)

There will be a huge market for local inference once it's cheap and widely available.

Try to imagine output token speeds of 15,000 tok/s and a time-to-first-token of 200ms. (This has already been done for Llama 8B.)

Now imagine gargantuan context windows (2M, 4M, or even bigger); keep in mind the 1M context windows were science fiction a few years ago... now imagine having this on a local model on something like a phone or portable device that can be gathering data about things you're doing and constantly run inference for things useful to you. An obvious example of this would be a chatbot you can talk to that responds like a normal human conversation and doesn't have delays, but that's just scratching the surface.

> There will be a huge market for local inference once it's cheap and widely available.

I've seen public pronouncements that the RAM shortage could persist for a decade.

And then if consider that the constraint on local LLMs isn't just memory size but bandwidth ...

If you take something like a DGX Spark and increase its memory to 512GB that doesn't even solve the problem. Because the bandwidth of DDR5 just can't manage reasonable speeds for decode. If you take a dense model or an MoE model uses up most of that 128GB in active decode you will only get like 15 tok/sec. "Real" datacentre inference boxes use high bandwidth memory that is 10x the speed.

I think we're unfortunately a long way off, unless people learn to accept working with much less intelligent models locally.

The innovation is going to have to come on the research & software side -- we need to find ways to pack more intelligence into a smaller number of parameters.

Isn’t that innovation precisely what Qwen did?

They are one of many working on this. It's obviously the holy grail.

RTX Spark does go up to 128gb No idea how it compares tho

I have one. The limitation (beyond total size of the memory) with the Spark is DDR5. "Real" inference hardware is HBM (high bandwidth memory) which is like 10x the performance.

So for prefill -- which is more about compute than bandwidth -- the Spark performs quite admirably. But on decode it's highly bandwidth constrained. Some smaller MoE models (e.g. Gemma4) can do 60-70 tok/second but anything dense, or anything that is actually filling up most of that 128GB is going to choke out around 15 tok/sec. Even at NVFP4.

My experiments with this are at https://github.com/rdaum/eider/

For my current work I get to log into trays on a real GB300. It's somewhat comical that NVIDIA is marketing the little baby on my shelf here as even in the same universe as that. Which is basically like having access to a super computer.

The individual-shaped-hardware problem gets even sharper at the phone end. Shipping a 3B model on-device, the usable RAM budget after the OS and everything else is more like 2-4GBtotal, not per-model so it's not 'can I afford more VRAM', it's 'can I fit a language model and an STT model and embeddings without the OS killing my process'. Feels like phones are the most hardware-constrained 'individual' tier and get the least airtime in these kind of discussion. Is that because the models that fit are still too limited to be interesting, or something else?

Recently spent a few hours messing with bonsai 27B and ternary bonsai 27B at total weights + context fitting in slightly under 6GB RAM, and it's just dumb as hell. It writes what seems like grammatically correct content but it's extremely limited.

It will also happily hallucinate new names and content to fill in gaps in its knowledge, and present the hallucations in what looks like a correctly formatted sentence, so it could fool a person who doesn't know the subject matter. Like, I asked it for a description of Seattle and it hallucinated a name and description of a nonexistant tallest building in the city and suggested the view from its observation deck .

https://prismml.com/news/bonsai-27b

https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf

In no way was I surprised, it's asking a lot of under 6GB RAM usage. But I think for 99% of people they will get better results doing something over the network where the weights and inference engine are not on the device.

Reply on news.ycombinator.com