You can do it on CPU now.

Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34