I've tried the 7B model with 32GiB of RAM (and plenty of swap) but my 10th gen Intel CPU just doesn't seem up to the task. For some reason, the CPU based libraries only seem to use a single thread and it takes forever to get any output.
with llama.cpp, you might need to pass in -t to set the thread count. What kind of OS / host environment are you using? I noticed very little speedup with using t=16 and t=32, it's possible the code simply hasn't been tested with such high core counts, or it's bumping into some structural limitation of how llama.cpp is implemented
I'm wondering if there might be a problem with your compiler setup? Do set -t to use more threads. I don't see improvement past the number of real (not virtual) cores. But I'm seeing about 100ms/token for 7B with -t 8.