Hacker News Reader

Does this mean us plebs can run LLMs on gimped VRAM Nvidia lower end cards?

I don't think so. It seems to just lower the ram needed for the context window. Not for loading the model on the vram.