I'm developing framework [1] in Golang with this goal in mind :) It successfully runs relatively big LLM right now, and diffusion models will be the next step

[1] https://github.com/gotzmann/llama.go/

Yes, you can run inference at decent speeds on CPU with llama.cpp. A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issues/34

There a lot of optimizations that can be done. Here's one w/ potentially a 15X AVX speedup for example: https://github.com/ggerganov/llama.cpp/pull/996

quantized Vicuna runs ok-ish in my 16GB i7 laptop (onboard graphics) and the output is usable

see this comparison: https://old.reddit.com/r/LocalLLaMA/comments/12ezcly/compari...

these models quantised to 4bit should run in CPU set ups with 16GB of RAM + 16GB of swap (Linux) and perhaps other setups run similarly

I've run LLaMa models on my CPU before, ViT-L and the Q-former are two transformer models as well, so I can't see why they wouldn't run on a CPU.

Someone is probably going to port it to llama.cpp soon.

It will do, probably quite soon. Many people are trying.