Hacker News Reader

Comment by chrishynes | Hacker News Reader

Why can't this scale to run much larger models on CPU backed by flash with good access patterns?

Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD.

I think they got something like 10 _seconds per token_ (not tokens per second).

EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri

That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient.

As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like.

[0] https://shir-man.com/tokens-per-second/?speed=1.8

I used to run Povray as a kid by setting up a scene in the evening and then letting it render all night and admiring the output in the morning.

I could see where you could set up some coding task and let it churn all night.

So while SSD streaming is interesting I'm not sure it's exactly the same thing as the per-layer embedding that is being utilized in tandem with streaming here. To utilize per-layer embedding, it would have had to be trained that way, which GLM 5.2 was not.

Rig with 6x RTX 5090. I would not say it is RAM only

Unless I am mis-reading the benchmarks, some of those are run as CPU only, no GPU involvement.

more interesting would be using some kind of FPGA to logic glue each RAM socket interface bitplane to a hard drive (so a collection of hard drives with ridiculous collective bandwidth). Perhaps a single RAM socket contains actual RAM and the linux kernel would have to be modified to only use the real RAM memory region for OS and inference software, with the inference software rewritten to stream LLM weights deterministically from the hijacked RAM slot physical memory regions. Obviously the FPGA can't truly achieve the CAS latencies over the HDD (unless the HDD firmware was rewritten so it can predict the next deterministic token sufficiently in advance to cache the result and stream it just in time to FPGA then "RAM" socket...) but even if the HDD firmware can't be reprogrammed for some reason, the FPGA knows what memory address will be deterministically fetched next, so it can make the requests to the parallel array of HDD's ahead of time.

My guess is because the ESP32's flash is only ~1/4 the bandwidth of the internal SRAM. If you do this on a more powerful system not only is the gap much wider but you also have much more compute you need to keep fed with bandwidth to be efficient.

It's also mapped into the address space so there's very little extra latency in grabbing the embedding as opposed to something like nvme that will have to setup a command list, submit it to the drive's microcontroller, wait for the op to be processed, etc.

Reply on news.ycombinator.com