Comment by horsawlarway | Hacker News Reader

Comment by horsawlarway | Hacker News Reader

Parent Full thread

For personal use, yes.

I replaced a $100/m subscription to claude in favor of running pi harness pointed at unsloth studio, using both qwen (unsloth/Qwen3.6-35B-A3B-MTP-GGUF) and gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models, depending on my mood.

I have a machine I built about 5 years ago with dual RTX3090s in it (I was going to build a new gaming machine anyways, and the llama release had just dropped so I tacked another used 3090 onto the build), and I get ~150tok/s on either of those models (at UD-Q4_K_XL quant) and can use the entire 300k context length without having to exit VRAM.

To be very clear - it's not as good as claude. But it's free and not so much worse that it matters significantly.

For my personal needs, free beats $100/m.

I also have an openclaw instance pointed at the same inference server, and it's great for that (genuinely solid use-case for local models).

Some example projects

- Replacement launcher for android tvs (with usage monitoring and tracking for kids)

- Custom admin portals for my k8s cluster services

- Custom home assistant integrations/automations (recently some shelly devices for power monitoring and switching)

- Grocery list management and meal planning (mostly via openclaw)

- some custom workflows for 3d asset generation in comfyui.

---

Long story short, if you're trying to make money via software... I'd probably still recommend using a paid provider. But the local models are very capable of cool stuff.

2x RTX3090 are around $4400. Without any electricity costs or other parts, that's 3.6 years of $100/m claude.

Assuming the $100/m claude subscription is still around in three years.

we will be lucky if it's still around in 3 months..

I think there’s a reasonable argument that a burst bubble will cause prices to drop. Prices are very high because they’re trying to justify these trillion dollar valuations on IP alone. If that fantasy goes away then prices will fall down to just silicon and electricity, which looks more like Chinese model prices. Hard to say how it will play out but the direction isn’t obvious to me.

Yes, today is not a great time to purchase hardware.

When I bought, I paid $850 a piece. And I needed one anyways for the gaming I was going to do.

My guess is the next good time to buy is going to be 24-36 months from now, depending on how the AI bubble goes.

---

I'll add to this, I personally don't like Apple hardware (not so much related to the hardware as their company philosophy) but their machines with unified memory (or AMDs latest unified memory offerings) get pretty equivalent speeds to my 3090s, and are probably a much better modern entrypoint to local llms.

There's a reason the joke is that Silicon Valley software devs bought up all the Mac minis for OpenClaw.

You can get a 48gb unified RAM M4 pro mac mini for ~2k. If you're not going to do much else with the machine, it's what I'd pick as my budget inference device right now. Spend a year of claude now, get ~150tok/s for the next decade (plus) for ~free.

If you want more capable and are willing to spend a little more, go with the newer Ryzen AI Max+ 395 machines.

You'll spend less on power too.

My last suggestion would be to go buy an RTX3090 at this point. You can do a lot better for a lot cheaper.

If you're willing to go the AMD route, the AMD Radeon Pro R9700 definitely looks interesting for the price compared to NVidia.

Can we also run LLMs on Radeon?

I run qwen 27B:Q4 @ 130k context at 50 t/s on a single R9700, and have a 7900XT that runs mellum 12B:Q8 as its subagent. R9700s do really well at low wattage and underclocking as well. It's designed to run at 300W, mine is throttled at 210W, and only had an 8% slowdown. If I had somewhere else to put my desktop in my house, I'd bump it up to 240W and there would be zero perf degradation.

That's also years of top tier PC gaming, if you're into that.

2x RTX3090 is extremely overkill for gaming, you can run any released game on earth on ultra for much less

1x RTX3090 is absolutely not overkill for gaming however. Nowadays it's barely enough to get 60FPS in 4K in some recently released games. But the shocking part is that my 3090 is still probably worth as much as when I bought it about 4 years ago.

It's probably worth more now.

Having a second card doesn't really work well for gaming.

I can’t run 4k HDR cyberpunk 2077 at 240hz with path tracing. I’m managing ~120fps. I’ve got a Blackwell 6000. I didn’t buy it for games, but there are still games and setups where the GPU is the bottleneck. I don’t even have an 8k TV.

There is currently no gpu in production that can max out the largest and fastest displays in graphically demanding games. We have monitors that are the equivalent of two 4k monitors side by side and run at 240hz. I have a 5080 and have to turn down settings to get 60fps in cyberpunk.

What if you do integer downscaling to 1080p on those 4k displays?

That's how most raytracing is done these days anyway. The game is rendered at a much lower resolution, the raytracing math is applied, and then it is upsampled to the target resolution.

If you set the target resolution to 1080p, not much changes in the render pipeline except the that final upscaling step. To get better quality, the lower resolution is bumped up so there is more data to work with for the upsampling, but the scaling performance can be very hit or miss depending on the game as the engine itself often can play a huge role in rendering performance.

As far as rendering the 1080p image at 4k, yea it works fine, but there will always be little artefacts that remain for those looking for them. 1440p seems to be the sweet spot for gamers today, but 4k is really nice for when you're not gaming as most online video is now made for dual use on televisions.

what?

AFAIK nvidia cards dont work in tandem (aka sli in the past) very well these days. So that aint true.

Also, 2 gens old means bad performance at ray tracing, abysmal path tracing if at all. Pretty sure it can't run smoothly CP2077 in native 4k without dlss upscalers with all on ultra.

You can have the 2nd card as an offload for upscaling, frame generation and whatnot.

When I'm not running models I use the 2nd one in a pass-thru configuration to a windows vm for various things, usually gaming.

Those GPUs can also play video games or mine cryptocurrency. They can also be sold later.

We should own things, not rent them. We should all do what we can to keep the fabled 2030 agenda at bay.

If you pay $2200 for a 3090 you're a sucker. They're not worth anything close to that.

Or a really excellent experience playing Satisfactory with the settings cranked up, which is priceless.

Christ GPU prices have gotten crazy

How do AMD cards perform with LLMs? A 9070 is sold for ~$600 and has 16GB VRAM

In my personal experience, I wouldn't bother with 16GB cards for coding -- the useful models are _slightly_ too large to work at any reasonable speed

That's not my experience, and the trajectory is good anyway - what doesn't work perfectly today will be just fine in a few months.

In a quickly moving field, it's amazing how much money one can save by overcoming FOMO and not living on the bleeding edge. It's like waiting for Steam sales, the games will be just as good.

Curious what model you're using that works well on a 16GB card? I very much want to use my 5080 for inference, but everything I've tried so far has either just not been good enough or painfully slow.

Qwen 3.5-9b-Q4_K_M.

I have a 5080 too! For me, the key has been dropping Ollama for Llama.cpp, which is not particularly scary to configure anymore and just skyrocketed performance. I download the models with LM Studio, then run them with llama.cpp.

gemma 4 12b

This can work for some things but I wanted to run hermes with 16gb but 12b is too low and the context was too limited, they recommend 27b

That should do pretty well. Memory bandwidth is the biggest bottleneck for token generation, at 644 GB/s you should be able to do pretty well on a 9070, while prompt proessing is more compute bound and Nvidia tends to have the edge there.

16 GiB won't fit you much, so you'd probably want at least 2x, and preferably 3x of those, and then you need a motherboard, power, etc. that can handle that.

You can get an R9700 with 32gb vram for ~$1200-1400 depending on where you live, which is probably a better option for AI use than 2x 9070(xt)

Yeah, definitely.

My 3090 was $700 shipped. Shop around. I'd expect to pay "thousands" for the nodded 3090 with 48+ gigs of vram.

It was ~8mo ago.

Not anymore.

I cheaped out and got 2 7900XT I get about 80 tps on qwen3.6 35b a3b. The cost when I got them before the memory crunch was $1400ish. On retrospect I should have forked over a couple hundred more and gotten the 7900XTX for the extra VRAM.

You can get 60tps with three 1080tis and the sparse model, and I bet two 16gb 5060tis would do the same for ~1200. One 3090 is enough for a useful system, even on an old am4 host.

Dual 5060ti 16gb does over 100 tok/s on 35B A3B. Even with PCIE Gen 4 x4, which quite a lot of motherboards can do. Though Gen 4 x8 or Gen 5 x4 is slightly faster. Misc working notes on this hardware combo here, https://github.com/jonnor/embeddedml/tree/master/handson/mic...

Look in the used market, not new. There must some that can be had for much, much less than that.

In 3.6 years, chances are they are still worth $3k. Unless some new chip fab pops up that can spam the chip market. Even if the AI bubble bursts, I doubt we'll see high-RAM GPUs sell off.

> gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models

Since you're running quantized (at UD-Q4_K_XL) , check out the "qat" models (unsloth/gemma-4-26B-A4B-it-qat-GGUF) !

- https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF (With "Jun 9 Update: Added MTP support.")

- https://blog.google/innovation-and-ai/technology/developers-...

TIL:

> Quantization-Aware Training (QAT) [...] allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model

How is the the QAT models at coding? I looked for opinions since the release and haven't found much.

> unsloth/Qwen3.6-35B-A3B-MTP-GGUF

I've actually tried this exact same model locally as well.. albeit on just a single 3090 at 128k context and I got around 40-60tok/s with Q4_K quantization.

The thing that bugged me the most was really the quality of the output on moderately complex real-world coding tasks. Having to switch between "prompt/vibe" and "manually implement" is such a big context switch burden, because you really have to ask yourself every few minutes if you're "holding it wrong" or the model is just too stupid.

It also doesn't really seem to handle transitions from "low-level implementation detail" to "high-level design" well, e.g., it wouldn't easily render tables and such. With Claude I don't have this issue.. so I think for now my verdict would be that it's not really a viable replacement. I really hope it will be in a few months time.

Oh and I used "aider" to replace claude CLI, which maybe that's also sub-optimal.. I'm not sure. The MCP marketplaces are useful of course, though arguably you could just manually replace them over time.

I don't generally switch to implementing myself on the model, although there are definitely times where I stop it and correct it mid-task.

It's prone to thinking longer and more repetitively, again - it's definitely not opus 4.7/4.8.

I've been using pi.dev as my harness for it, and been pleasantly surprised by how nice it feels (I have used aider, but only very briefly and quite a while back - so I can't realistically compare).

I would say it's roughly where I felt claude was a year back - Most of the sessions need to be more "pair programming" and less "I let it run for hours".

I'm a big fan of frequent "human in the loop" style workflows even when I'm on something like opus at work, though. I have opinions about lots of things, and re-inforcing that the model should stop and ask frequently seems to get me considerably better output, without having to "re-roll" if you will.

I've done a good bit of management, and I think it's roughly producing what a junior dev might produce in a day every 5 minutes. And just like a junior dev, you need to be steering it back on track fairly often.

Opus feels more like a mid-level at this point. I can hand it a chunk of work and "leave" but I still get better output if I'm checked-in and watching/steering.

I'm so out of the loop on this stuff, it's the first time in my IT career I feel really behind on things.

I've used Claude Opus to quickly and effectively pound out some 100-200 line scripts that integrate with a vendor's API, and it one-shotted them both almost perfectly.

I wonder if for a lot of these local models, the scope of the AI assistance should simply be smaller: You architect the tools and the function definitions, and then tell AI to implement one at a time? Does anyone do that rigorously?

100-200 like scripts are tiny especially for something easy to scope like a vendor api. Give opus a much, much larger challenge and see what you get back. You really don’t need to see the code much at all anymore except for some steering now and then.

See, that makes it sound better in my world: I'm doing all 'manually implement' all the time, and have no interest in becoming the ten millionth manager to hit the software dev landscape. It boggles my mind that people think this is a win.

I regularly use a pocket calculator: either a physical one, or Apple's Calculator app if I want more decimal places. 'Too stupid' isn't a thing for me if it can cough up some math that would be inconvenient for me to work out by hand. tok/s also isn't a concern because I'm not expecting more than I can read. My ideal scenario would be occasional diversions into querying a 'coding calculator' that can give examples along the lines I want, my way.

I'll make a mental note that Qwen shows signs of being the kind of calculator I'd use for a specific task. Context switching isn't a burden if you're not looking to switch over to vibe/manage and stay there.

The things is: it does _feel_ like you're moving faster when Claude is in the zone and does what it's supposed to. You're essentially flying a plane on auto pilot, occasionally telling it to slightly adjust course. Only that now you can fly 10 planes in parallel, all to different destinations.

Is it _objectively_ more productive? I doubt there's a clear-cut answer in the long run (my main suspicion is that since you're essentially creating 10x unnecessary complexity, you'll likely never recover from all the cruft and maintenance kills you in the end - maybe people will find solutions for that though).

aider sucks tbh... you should invest time in learning how to customise pi. every other harness is crap and hype.

Thanks! yeah, maybe I'll try to look at that next..

Did you double the tokens per second by adding a second GPU or was the increase significantly less?

No real change in inference speed. It basically just allows me to slot in more context or a bigger model.

A single RTX-3090 will do approximately the same tok/s, but it won't fit the entire 300k context in VRAM.

Sometimes that matters, a lot of times it doesn't.

On the speed front - MOE models are great. Biggest perf difference in modern models is the move to MOE architectures.

I get very similar quality from the both the Gemma-4 31B dense model, and the Gemma-4 26B MOE model (both at Q4 quant) but the MOE version runs at ~3 times the speed (150tok/s vs 46tok/s).

You’re adding extra gpu for more vram, not speed.

Mind sharing your setup? I also have dual 3090s, but getting nowhere close to 300k context limits with 4 bit quantized models at that size (using vllm).

That sounds amazing. If I had some GPUs sitting around, I would totally do it. Sounds expensive to do it otherwise though.

Grocery list management and meal planning sounds interesting. Would you mind sharing a little bit more on your use case please?

Reply on news.ycombinator.com