Comment by ak_t | Hacker News Reader

Comment by ak_t | Hacker News Reader

Parent Full thread

Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago.

I've been running this model locally for a week, and the preview version before that. This updated one feels like a whole tier up. It's very capable for debugging and analyzing documents/data I upload.

The killer feature, IMO, is the speed. On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. I saw 1000 tok/s with ~64 concurrent streams on vLLM.

That's fast enough that you can interactively chat with it without switching tabs while you wait, and its a ~300B (13B active, hence the speed) model so the responses are also very good. It's actually more convenient now for me to direct 95%+ of my day to day usage to my local model, and only use Claude Fable for really big coding tasks.

Until this model was released, I was contemplating spending even more money on hardware to run GLM5.2 (~750B) at reasonable speeds, but I no longer feel that need. This is smart enough, and I think it only gets much better for local models from here.

What quantization level is that? Because official endpoints are slow.

It doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.

V4 Flash fits entirely in two RTX Pro 6000s without any quantization at all.

From opencode go $10/mo plan I get between 60 t/s and 100 token/s even with large contexts of 150k+ tokens.

I wouldn't call 80 t/s slow.

You are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.

That's across 64 concurrent streams; you could make more concurrent requests to DeepSeek API no?

I get around 60-110 TPS on tensorx.ai hosted models.

What runtime are you using with the 2x RTX Pro 6000 Blackwell machine? I have the same setup and tried DSv4 Flash on vLLM and ran into a ton of kernel bugs that don't seem to have been fixed yet.

so..you guys have $20k+ rigs?

This one works great. https://github.com/ormandj/sglang-deepseek-v4-flash-sm120

This also works great: https://github.com/antirez/ds4

> On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream.

For reference, on a 1x B300 it's over 400 tok/s decode on a single stream.

I wish I could say that it performs reasonably on my hardware. 8x Radeon AI Pro 9700XTs, and I can't get it to hit double-digit tokens per second. Neither vllm nor llama.cpp, with various combinations of quants, draft models, and parallelism methods can get it to run at a tolerable speed. I'll be sticking with StepFun 3.7-Flash for the foreseeable future :(

I'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and usually for highly parallel loads. Any tips on where I should look first for an obvious blunder?

I'm guessing tensor parallelism or similar?

You're gonna want a custom vLLM build.

Here's a runbook: https://github.com/local-inference-lab/rtx6kpro/blob/master/...

If the newer builds aren't working, you might try running the old v6 build (based on the eldritch-enlightenment image). gilded-gnosis gave me some problems that I haven't bothered to track down, the old builds are still gonna blow away llama-server performance. And that's before you get hooked on vLLM's PagedAttention and can run multiple sequences without a ton of extra overhead.

Reply on news.ycombinator.com