Hacker News Reader

You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.

View on HN

It would apply equally to GPU or RAM inference as those are also bandwidth constrained on decode, so people already try to optimize for it.

surely the supply of unified memory will rise to meet demand before this is needed

Reply on news.ycombinator.com