Hacker News Reader

Comment by wizee | Hacker News Reader

Qwen 3.6 27B is quite good for agentic coding, and practical to run on consumer hardware. You need a system with either 32+ GB VRAM, or a unified memory system with 48+ GB VRAM and a decent integrated GPU. While not cheap, such a setup is still attainable for much of the world, and will eventually get cheaper over time. Open models hosted on non-American clouds also remain an option with a much lower barrier to entry, for cases where privacy is less critical.

View on HN

There was an article on HN a few weeks ago where someone detailed how they managed to get an old datacenter GPU to run in their consumer PC, getting decent performance with qwen. He spent something like $200 on the GPU (second hand of course).

So yeah, I think models on local hardware will be quite common soon among the tech savvy (such as people creating software).

Especially considering the millions of 2026-class data center GPUs that massively overinvested companies are currently buying, which will be obsolete in a few years.

I think those are going to be run until they die. The capex vs opex is too high to obsolete them in a few years. They'll keep serving current gen LLMs for as long as they keep running.

They can also be used for other things than running the main frontier whatever model as well.

E.g. grok isn't truly multi-modal, it has a callable tool that is a separate VLM it invokes on image URLs or files (for a long time it was grok-1.5v, but I think they have upgraded now, it was pretty bad).

And then you have the small summarizer models for the CoT/thought traces, the guidable summarizer models for the standard browse tools, etc.

There's a ton of stuff that can use an aging GPU.

Yes, sure, but not efficiently. Even Pops will not want to run four hair dyer GPUs 24-7 in the garage.

It won't make sense to run them after two years. The vendors will be limited on datacenter space, power and cooling, and there will be new hardware available that will run the same models at a fraction of the power.

A100 -> H100 was >3x tokens per joule, H100 -> B200 >10x. There are significant low-hanging fruit still available in architectural efficiency, and the vendors are chasing them.

This is the big risk for AI companies that I feel is not being sufficiently priced in. Almost none of the investments they are making are durable, the depreciation schedules for everything but the real estate should be less than 24 months. Until the hardware is stable enough that you only get double-digit % improvements per generation, it should almost be counted as opex.

The annual operating cost of these is <10% purchase price annually. Even if the B200 is 10x more efficient in practice you can still operate the H100 at profit.

As it stands there's way more demand than supply. The new GPUs are going to run frontier models while the older ones serve smaller ones.

That said some of these are running in tents hooked up to mobile turbines. I can see some of those going away but generally I think you'll see them used until they start to fail in 5-10 years.

H100 were released in Oct 2022. They are now more expensive than at release time.

I think you are right when you factor in the much more efficient newer high end GPU. That is what may make the current GPu investments obsolete in 2-4 years.

aiui nvidia has buyback agreements covering these to prevent them from being sold off

Indeed, and with some tinkering around the harness it can even punch way above its weight.

I've seen folks make it work with a 3090 on 4 bit quant using turboquant for KV cache. That's key because 3090s remain the most cost effective gpu metal for enthusiasts (albeit 24g) and the jump to 5090 (32g) is quite expensive and not always worth the LLM specific performance; sadly, good 32g metal is somewhat lacking in the price point at or above the 3090.

> You need a system with either 32+ GB VRAM

I do hope you're right that it will get cheaper over time (it should), but right now 32GB of VRAM is not affordable to a lot of people. You're talking ~$4500 just for the GPU, or $800 ish used if you can find one.

For inference you can split the 32GB between two 16GB cards. Two new 5060tis for ~€1000 in total is more than fine.

It's a tad less efficient and a bit more of a hassle, but still a good experience for only a fraction of the price.

A Mac laptop can be had with 32GB of RAM for far less than $4500. Not sure if they actually need 32GB of discreet GPU RAM. My Mac laptop does run Qwen at a reasonable speed.

Intel Arc Pro B70 32gb for $999ish https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/...

Reply on news.ycombinator.com