Hacker News Reader

Comment by a11r | Hacker News Reader

I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK

View on HN

I just ran a set of benchmarks, ninfer-3090-qwen3.8-27b (so mix of Q4 and Q5) vs strata-qwen3.8-flash-next-iq3_xxs (so Q3):

│------------------- │ Ninfer-3090 │ Strata

│ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%)

│ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%)

│ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)

│ API failures------ │ 10 │ 5

- Ninfer generation: ~122 min total.

- Strata generation: ~142 min total.

So strata is a little slower, but keep in mind that ninfer-3090 is very optimized for a Qwen 3.8. Standard Qwen 3.8 runs at 20 t/s, this modified version can do 50 t/s (but it's extremely long in it's thinking, it just goes on and on.

This is on a 3090 that will crash unless power capped, with a Zen 2 CPU, 64GB DDR4 with a PCIe that refuses to go higher than 8x (basically pretty crappy all in all).

Yet with some tweaking and optimizing I still manage to get strata to run at 40 to 60 t/s.

That strata has been optimized on my Oh My Pi conversations. So when I'm using it, it's probably faster and closer to ninfer in speed than during those unoptimized benchmark tests.

Anecdotally, I've found that 27B at 4bit hallucinates a lot more than QFN at 3_xxs, particularly for less technical reasoning.

I've tried to use both for online comparison shopping. QFN not only seemed less delusional but also made useful observations and problem solved ways around many different website access issues.

You can use 4 spaces on HN to get code/monospaced formatting:

    │------------------- │ Ninfer-3090    │ Strata    
    │ Code generation    │ 52/78  (66.7%) │ 70/78   (89.7%)
    │ Code completion    │ 40/50  (80.0%) │ 44/50   (88.0%)
    │ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)
    │ API failures------ │ 10             │ 5

In my private C coding benchmark, Strata's Qwen Flash Next Coder IQ1_M beats Luna low and runs at 125 tok/s on my 5090

Not as fast for me. I was able to snatch a spark before the horrible price hike, and I have it at 40/50tok/sec. Before I had the great 3090 setup which is what led me to get the spark. That 3090 is now relegated to creating videos of my kids doing dumb things.

I now backordered two more sparks, before the hike... hoping they will honor the agreed price and dont cancel on me. They should arrive in a month,

Its evidently clear the frontier labs won't keep on giving us cheap inference for much longer. I also watch in disbelief how people say a sub is cheaper, which probably is. But you loose on so many other things (privacy, predictability, censorship...)

> it's extremely long in it's thinking, it just goes on and on

Known issue with this model, I recommend setting thinking to 'medium' instead of default 'xhigh'.

Ill check for this in my setup. I thought it came with only one thinking level causing the “meandering” as it refers to its own long CoT.

I’m doubting the quality of the model, even comparing with luna 5.6, it is giving such bracingly wrong answers. It claims other tools have changed the code when pressed why it made a mistake. That code or tool didn’t come close to working on the project. It is scary.

Unrelated... just use your just locally installed qwen 3.8 and instruct it to figure out why you arent at 16x.

For me, I didnt know I was on 4x and I was able to, giving it enough permissions, with a good harness like PI, to figure things out and go 4x -> 8x -> 16x.

It probably will take some opening the case and switching ssds around, but it's amazing how powerful these models have become.

For the 1 dollar per hour, may I ask (without trying to incite things) why you would not just buy a ChatGPT Plus subscription or other 20 dollar subscriptions? Do you like the privacy?

maybe ethics as well. maybe people like sleeping well at night.

Those plans are super slow right now. Nowhere near 100t/s.

I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot

I was firmly wanting to move towards local llm, and since the Sonnet 5.5 and recent Opus speed increases, I'm putting that on hold until local inference speeds can be improved. Not sure how much juice there is to squeeze there, but I'm hopeful.

While this may be true, their throughput is practically unbounded. this has been a primary challenge to using local models at work, where its normal for me to have 1-6 sessions churning away at once. So while 100t/s on a single session is very fast, theres no way you could get a cumulative ~500t/s from a consumer setup like this, do the bandwitdth bottleneck on a single card/cpu/ram/ssd (tends to be gpu ram bandwidth is the limiter in my setup, but they all have respective limiters to this kind of throughput)

Does it matter if those 100 tokens per second are way crappier?

Access to models is currently being priced below it's actual value. Largely because the market hasn't settled yet. Once it does the prices to access AI will increase -signifigantly.

It will be less of an issue for people who have already got the equipment and the practice running locally than it will be for people who have simply been relying on ChatGPT.

Prices on openrouter are not subsidized and are very affordable.

People think claude costs $200/month, it actually costs $200/month and all your labor to train their models, improve their product, and all your best ideas and work that they will feed back into their company. Its very expensive

You can opt out of model training with a checkbox, though I doubt most companies respect this completely. At the very least, I imagine they find loopholes to exploit.

But on the other hand I'm giving them my best and worst ideas and the value I get in return outweighs what I'm contributing to them. I also don't have the problems of initial HW cost and maintenance.

> You can opt out of model training with a checkbox

Do you trust that a company who can't even do basic IT ops monitoring are capable of respecting a "no training checkbox"?

Great, then they will PII-scrub my sessions before feeding them into the training pool :-)

Just as curiosity, how long does it take to set the environment up and running?

Is it viable to start/stop it multiple times per day?

Yes, you can probably get the whole thing up and running in about an hour the first time. If you pause and restart, it takes about 15 minutes to load the models from disk into GPU memory, so budget for cold startup time.

How does it take fifteen minutes to read <100 GB into GPU memory? Shouldn't that be limited by SSD speed with everything slower than a minute being a terrible ssd?

A lot of cloud platforms have terrible slow network storage. They also might need to compile the GPU kernels fresh as they might not have a persistent CUDA cache.

It also depends on the runtime, vllm is unbelievably slow at model loading compared to llama.cpp

this is not right. vllm can easily load safetensors model at >5GB/s if not faster when setup right. did you make the compile cache persist? if you use docker, you should bind mount the kernel compile cache, so they don't need to be recompiled each time vllm restart.

I just had claude code set it up for me and create some startup scripts. The longest part was the model download.

Can you say more about where you're renting the RTX Pro 6000 for $1/hour?

vast.ai?

I am renting spot VMs from Nebius. I've tried a variety of other providers like Vast and Spheron. Vast worked well for renting 5090s but I like the large memory and pricing I'm getting at Nebius for RTX Pro 6000. The extra RAM really matters because I need PLE to offload the ngram to RAM.

ISTA-DASLab's IQ3_S quant is really good https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

What are you running it on? I'm getting mixed results using the IQ3_XXS quant which supposedly matches baseline, it feels significantly degraded.

I've run DeepSeek V4 Flash on DS4 on single DGX and standard model weights on aDGX cluster. There was some degradation going to the hybrid 2 but quant but really not much.

The larger the model, the more you can go down. K3 in Q1 will match and likely beat Qwen3.8-Flash-Next.

$1/hr sounds great. Where are you getting it for those prices?

[flagged]

No? Per someone else's comment, presumably they're using vast.ai which does indeed list the 6000 for $1/hr

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Apologies. I had recently read (I think from someone on here) that they were in some discord where those sellers advertise. Whoever it was mentioned that the service was incredible value for what they got, but admittedly it was for toy projects, there was no written agreements, and uptime was something like 98%. I should have found the source and linked, and will try to do better overall in my replies!

$1/hour ? I need that deal as well.

Not the original commenter, but Upcloud has them at that price for spot instances: https://upcloud.com/pricing/

That's cheap, too. Which hosting service are you using?

Reply on news.ycombinator.com