Hacker News Reader

8B models are pretty fast even on something like a 3060 depending on deployment method (for example Q4 on Ollama).

They're fast enough for me on CPU even.

What CPU and memory specs do you have?

Sure, except those smaller models are only useful for some novelty and creative tasks. Give them a programming or logic problem and they fall flat on their face.

As I mentioned in a comment upthread, I find ~30B models to be the minimum for getting somewhat reliable output. Though even ~70B models pale in comparison with popular cloud LLMs. Local LLMs just can't compete with the quality of cloud services, so they're not worth using for most professional tasks.

They might be talking about 70B

Wait, is 70b considered on the smaller side these days?

Relative to deepseek's latest, yeah