Hacker News Reader

Comment by tarpitt | Hacker News Reader

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2

I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.

>SSD Wear Warning

> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.

Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?

View on HN

I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!

I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.

I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.

If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.

I'm going to benchmark this right now with what I have and I'll get back to you on github.

Really thanks!!

At least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.

If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.

> OS page cache can generate writes

Is this a hallucination? What am I missing? Why would heavy reads generate writes?

Spilling

Wouldn’t turning off swap fix this issue?

Better to just change swapiness?

https://askubuntu.com/questions/103915/how-do-i-configure-sw...

> Is this a hallucination? What am I missing? Why would heavy reads generate writes?

I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?

I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?

Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.

That's possibly a good idea! We can work on it!

I also just edited my comment with more ideas in the beginning, sorry

Reply on news.ycombinator.com