Hacker News Reader

"Yes", but it's going to be unbearably slow on CPU without some form of GPU acceleration.

You'll probably get more info/responses from https://www.reddit.com/r/LocalLLaMA/

Using llama.cpp on a 128 GB server running codellama-70b-python.Q4_K_M.gguf I get 1.3 tokens per second which is too slow. With Nous-Hermes-2-Mistral-7B-DPO.Q5_K_M.gguf I get 8.3 tokens per second which is usable.

Reply on news.ycombinator.com