Hacker News Reader

Comment by brcmthrowaway | Hacker News Reader

Is there a gamechanger around the corner to reduce DRAM requirements?

I can imagine someone building a device that uses NAND flash or similar tech but with a very different controller that is optimized for streaming the data out with a predetermined access pattern at very high speed.

Flash, like pretty much every solid state storage technology, can scale its output bandwidth to ridiculous levels limited pretty much only by the readout circuitry. There may be a price to pay in power consumption, though.

It’s called “HBF”, high bandwidth flash, and it’s on its way!

You could always stream from SSD storage. Especially effective if you get a cheap old-gen HEDT with lots of PCIe slots to add NVMe storage to and reasonable overall PCIe bandwidth.

That nearly certainly boots you to secs-per-tok land (as opposed to tok/s). Plausible if you are willing to wait hours to days for responses for simple testing, but not (debatably) "usable".

Demonstrated inference speed for the biggest open-weight models is indeed around 1 token per second, with weights read from SSDs.

However, this is for relatively low-end systems, with a couple of fast SSDs providing around 20 GB/s throughput (or with a few more, but connected through relatively slow Thunderbolt, for a similar total throughput).

If you use 16-lane PCIe add-on cards with 4 M.2 slots for SSDs and a total throughput of 50 to 60 GB/s, you can quadruple the previous speed in a desktop PC where you use the GPU PCIe slot for SSDs (a fast CPU, e.g. an AMD 9950X, would be alone fast enough for inference limited by SSD throughput, so a discrete GPU would not be required).

If you have a server/workstation motherboard, e.g. with 6 16-lane PCIe slots, you might gain another factor of 4 in the inference speed, so one might reach around 15 tokens per second for a very big model, but the cost would also be high, with so many SSDs, even if at that number of SSDs each SSD could be the smallest that can be found with a PCIe 5.0 interface.

n-gram per-layer embeddings[1][2] might be it.

[1] https://sebastianraschka.com/llm-architecture-gallery/per-la...

[2]: See DS 4.1-Flash and Qwen-3.8-Next.

this is to offload VRAM to DRAM (for GP comment), and makes no difference for URAM

Am I missing a joke? WTF is URAM?

unified memory, not sure if anyone uses URAM, I human hallucinated it

You can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).

I was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!

Check out eugr’s TP=1 sparkrun recipe :)

It’s an NVFP4 quant, but it fits, and is surprisingly capable.

do you have a HF link? HF search is not uncovering it for me

(or is it somewhere else)

looks like this is likely it

https://github.com/spark-arena/eugr-recipes

https://github.com/eugr/spark-vllm-docker

https://github.com/spark-arena/eugr-recipes/blob/main/recipe...

This one!

I'd recommend pointing your agent at it (after installing sparkrun), and asking it to research the absolute latest in TP=1 Flash-Next - mine grabbed particular vLLM nightlies and mods to improve performance, and it was well worth it.

I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me. I'm not keen to run nightly vLLM, too many issues with it in the past. Going the qwen-next path means displacing things I use daily :/

I have a watchful eye on the diffusion ~ Jev/Kev PR

https://github.com/vllm-project/vllm/pull/57250

For what it's worth, Flash Next outperforms every other model that is available to us on the GB10 in all of my testing; though if you have two sparks then the TP=2 version is even better and easier (I don't think you'll need the nightly for that at all, just use the recipe)

I'm so tempted to buy a second one...

prices have gone up quite a bit...

I'm running embedding, reranking, and policy tuned models too, and a Jev/Kev when that's landed. Flash Next is not a substitute for those

I have OpenCode/Fireworks to access big models

Nah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)

interesting, peer comment seems to indicate this is a possibility as well, will have to take a deeper look

What kind of throughput do you see on what models?

check out the spark arena website, its the raison d'etre

GB10 boxes have way more compute than they have memory bandwidth, which nicely fits medium sized MoE models with speculative execution (MTP, DSpark/DFlash, etc)

Qwen 3.8 Flash Next (what I'm running basically entirely now) sees 30 / 35.0 / 45 tk/s for prose, analysis and code respectively for actual use (not short context benchmarking) with Pi. Thinking blocks are ~35tk/s or so.

The GB10 having so much compute is great for prefill too, 2000-3000/s for 14k to 64k token prompts (cold cache too) in the quick benchmark I did. 3500tk/s for warm cache which is nice :)

When I accidentally streamed my ngrams over the 2.5Gb/s network, it cut all the throughput down in half basically. Especially notable for the time-to-first-token, which is what clued me in that I'd messed up somehow!

For Qwen 3.8 27B, I got it up to a consistent 20tk-25tk/s but 27B thinks so much that it was honestly too painful: Flash Next is as smart, as useful, but much faster for real agentic dev usage IMO

Laguna S 2.1 saw similar numbers to Flash Next if I remember right, but their latest updates means it doesn't quite fit a GB10 128GB anymore at full context which is a shame.

Note: these are all NVFP4 quants (usually a dynamic one where some tensor layers are left at full precision though)

I personally stopped caring as much about the tok/s as the agents are largely in the background, and so have also moved preference from MoE to dense

I want to see about fine-tuning these models a bit on the GB10 to tame that over thinking and some other behaviors (like using tools I don't use)

qwen 3.8 seems to have been trained with some `rkt` that messes with tool outputs to "save tokens"

n-grams can be kept on SSD, no need to hold them in any kind of RAM (at least w/o batching)

Reply on news.ycombinator.com