I think using this project https://github.com/ggerganov/llama.cppav

on a CPU machine with AVX instructions would be a better bang for your buck than GPU. Depends on if your use case can tolerate the latency