In the article, Simon mentions the Q6_K.gguf model, which is about 40GB. A Mac Studio can handle this, but any of these models are going to be a tight fit or impossible on a Mac laptop without swapping to disk. Maybe NVME is fast enough that swapping isn't too terrible.
In my experience, the Mixtral models work pretty well on llama.cpp on my Linux workstation with a 10GB GPU, and offloading the rest to CPU.
It is impressive how fast the smaller models are improving. Still, a safe rule of thumb is the more RAM the better.
Also, really question how much you need to run these models locally. If you just want to play around with these models, it's probably far more cost effective to rent something in the cloud.
[1] https://github.com/sagemathinc/cocalc-howto/blob/main/ollama...
If you want to run most models, get 64GB. This just gives you some more room to work with.
If you want to run anything, get 128GB or more. Unquantized 70b? Check. Goliath 120b? Check.
Note that high end consumer gpus end at 24GB VRAM. I have one 7900xtx for running llms, and the best it can reliably run is 4-bit quantized 34b models, anything larger is partially in regular ram.
Is there anything useful you can do with 24 or 32GB of RAM with llms? Regular M2 Mac minis can only be ordered with up to 24GB of RAM. The Pro Mac mini M2 is upgradable to 32GB RAM.
I can have it run in on 'cpu' which is very slow, but offloading to the GPU runs out of memory.
Thanks a ton! I'm running on GPU w/ Mixtral 8x Instruct Q4_K_M now. tok/sec is about 4x what CPU only was. (Now at 26 tok/sec or so).
What I can draw from reading of that thread is that you can buy a Desktop Rig with 200GB memory bandwidth (comparable to m3 pro and max) and a lot of expansion capability (256GB RAM). You should find out if that's still good enough for your local use case for token per second or training.
Then just use SSH/XTerm(and possibly ngrok) to login with good speed from anywhere into your rig with a light M2 ?
16GB is not enough.
32GB is enough to run quantized Mixtral, which is the current best openly licensed model.
... but who knows what will emerge in the next 12 months?
I have 64GB and I'm regretting not shelling out for more.
Frustratingly you still have WAY more options for running interesting models on a Linux or Windows NVIDIA device, but I like Mac for a bunch of other reasons.
Maybe keep an eye out for M1 / M2 deals with high ram config? I've seen 64GB MBPs lately for <$2300 (slickdeals.net)
I think 32GB might be the best middle ground for my needs and budget constraints.
It's really a pity that you can't extend RAM in most Apple Silicon Macs and have to decide carefully upfront.
I currently run Mistral and a few mistral derivatives using Ollama with decent inference speed on a 2019 Intel Mac 32GB. So I assumed the new one with 32ish should do a better job.
I've tried vision model Llava as well, a bit more latency but works fine.
With Apple's own Mlx things might improve .
Bait aside, I'd love to read about how are you using those models. I'm mostly interested in code comprehension and meeting summarisation.
I'm going to bump up my usage of Mixtral a bit now to see how it feels for that kind of stuff.
Although for those napkin like ideas gpt4 (including the turbo variant) get costly quickly.
memory bandwidth is the key to model speed, and memory size is what enable you to use larger model (quantization let you push thing further, to a point) so one thing to note is that on the M3 pro/max only the top end model gets the full bandwidh, while the m1/m2 pro enjoy full bandwidth from a smaller memory size. this may be important if you value speed above model size or vice versa. M2 Pro, M2 Max get approximately 200 GB/s and 400 GB/s, but things are more complicated for m3: M3 Pro gets 150mb/s, and M3 max gets 300mb/s at 36gb and 400mb/s at 48gb
few more things to note:
it's absolutely fine to go and play around with llm but even with a llm monster machine there's nothing wrong in starting with smaller models and learning how to squeeze the maximum amount of work out of them. the learning do transfer to larger model. this may or may not be important if at some point you'll want to monetize or deploy to production what you learned. while the mac itself is a good investment for personal use, once you move to servers, cost skyrockets with model size, because of supply constraints on 40gb+ memory gpus. if you are dependent to a 70b parameter model, you'll have a hard time to make a cost effective solution. if it's stricly to playing around, you can disregard this concern
even if you're playing around, a 70b is going to run at 7 tokens / second, which is fine for a local chat, but if you are writing a program and need inference in bulk, it's fairly slow.
another thing of note is that while the field is still undecided on which size and architecture is good enough, the moltitude of small fish experimenting with tuning and mix of instructions are largely experimenting on smaller models. currently my favorite is openhermes-2.5-mistral-7b-16k, but it's not an indication that mistrals are strictly better than llama2, more an indication that experimenting with 7b is more cost effective for third parties without access to gpu than experimenting with 13b, and so you'll find 13b model kinda stagnating, with many of them trained in a period where people didn't really know the best parameters for finetuning and are so to say a bit behind the curve. a few tuners are working at 70b models, but these seems to be pivoting to mixtrals and the likes, which will cause a similar stagnation on the top end, that is, until llama3 or the next mixtral size drops, then, who knows