Hacker News Reader

> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models

I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?

Just ollama if you are feeling lazy. With ollama, pull the model (start with https://ollama.com/library/gemma3:27b), and `ollama run gemma3:27b`

If you want to build agentic frameworks, use llama.cpp with its built in http server, and build the framework with python