Hacker News Reader

Memory. I don't know the equation, but its very easy to see when you load a 128k context model at 8K vs 80K. The quant I am running would double VRAM requirements when loading 80K.

View on HN

This was my understanding too. Would love more people to chime in on the limits and costs of larger contexts.

Reply on news.ycombinator.com