Accelerators have nothing to do with it as we're mostly memory bound by HBM <> SRAM data transfer rather than compute bound.
Some of the things being done to improve quality of 6-8 bit inference use extra compute and push it a little in the other direction but it’s still pretty memory intense until the batch size gets quite large
Answer is - it is really works, but slow (about 0.5..1 tokens per second, with near 100% CPU usage).
i7-7700 is good weighted machine, but before I few times achieved memory speed bounds with highly optimized software. And it looks very different. When use all cores, I got somewhere about 50% of CPU usage.
BTW Llama.CPU is very good software.
Also, if you have just a single model you want to optimise (and not the training), you could build an array of asics that do specific matrix computations - then you don’t need to read weights from memory at all.