The financial aspects don’t work however: I can learn and experiment with what I have for local models, and I pay as I go on FireWorks.ai for open model inferencing and no matter how much I use this service my monthly bill is between $10 and $40 and much faster than any reasonable home rig.
Hybrid ‘small local’ and buying inference is the way I choose.
A 4 GPU linux box with 3090s, which are $1500 a piece right now, will blow this thing out of the water. Even 2x3090 rig will run most of the good local models like Gemma4:31b at 100+ tok/sec
The VRAM of the GPUs are MUCH faster than the unified ram within Apple Silicon. The only difference is the initial model load, which takes longer from disk to VRAM due to PCIE limitations, but once the model is loaded, GPUs can prefill and and generate tokens way faster than any Apple Silicon.
So is your desire to upgrade to mac because you just aren't aware of how to set up a GPU rig, or is it something else?
A PC with similar capabilities is going to sound like a jet taking off.
Not at all. Airflow with big fans is quiet. What I do hear is coil-whine. In fact my PC is quieter than my Macbook when both are running top speed. But one has 4000 AI TOPS.
* 3090 memory bandwidth: 936 GB/s
* Maximum NVLink bandwidth between two 3090s: ~56 GB/s one way
* Maximum PCIe v4 bandwidth between two 3090s: 31.5 GB/s one way
* M5 Ultra memory bandwidth: 1.2 TB/s
I know that memory bandwidth is only one factor influencing LLM performance, but this seems like a major problem if your goal is to run larger models that won't fit on one 3090 - and even those fitting on two 3090s with NVLink will be pretty restricted.
And, you have to include power. Where I am, the power for a rig with 2x 3090, running for four hours a day, would be around $70/month!
It's kinda like datacenter only exist to optimize compute cost through oversubscription of hardware/time share, cheaper business energy rates, and bulk discounts, compared to self hosting. ;)
You also don't need to run the models 24/7, nor is your computer gonna be doing inference all the time when you are coding.