I'm on phone and can't test this right now.
What are the system requirements for getting a decent model running in background?
Is it something I can run on a laptop with 8-12GB of RAM and not a huge harddrive?
What are the system requirements for getting a decent model running in background?
Is it something I can run on a laptop with 8-12GB of RAM and not a huge harddrive?
StableLM 3B ~4 GiB
You could go even lower with smaller quantization if necessary. I personally wouldn't use anything smaller than 7B and Mistral already pushing it in coherence. Overall it depends on your use case, not everyone needs smart models, or large context that sometimes takes half of required memory.
Codellama is also surprisingly good even for non-coding tasks