LLaMA 3 70B Llamafiles
huggingface.co
huggingface.co
in practice 10 tokens per second is kinda annoyingly slow
most local people would opt for a smaller 7b model
I’d be cautious of synthetic benchmarks, you never know how much of the scores are due to contamination or survivorship bias.
But for scrapping, and tool calling it represents a substantial gain.
If you want to run Q4_0 you'll probably be able to squeeze it on a $3,999.00 Macbook Pro M3 Max w/ 48GB of RAM.
If you want to run Q5_K_M or or Q8_0 the best choice is probably Mac Studio. I have an Apple M2 Ultra w/ 24‑core CPU, 60‑core GPU, 128GB RAM. It cost me $8000 with the monitor. If I run Meta-Llama-3-70B-Instruct.Q4_0.llamafile then I get 14 tok/sec (prompt eval is 82 tok/sec) thanks to the Metal GPU.
You could alternatively go on vast.ai and rent a system with 4x RTX 4090's for a few bucks an hour. That'll run 70b. Or you could build your own, but the graphics cards alone will cost $10k+.
AMD Threadripper Pro 7995WX ($10k) does a good job too. I get 5.9 tok/sec eval with Q4_0 and 49 tok/sec prompt. If I use F16 weights then prompt eval goes 65 tok/sec.
For comparison, I tried the same rig with a single RTX 4090, but it generated an abysmal 2.4 tokens/s, which I would consider nigh unusable. The difference between all GPU VRAM is significant.
If you’re frugal and don’t buy the latest and greatest CPUs, you can get by on a setup costing less than $1500, or less than about $2300 with two GPUs. Make sure you get a beefy power supply, though you can throttle the 3090s down to ~225W each and not lose much performance.
With a single 3090:
* Roughly half of the model layers will handled by the GPU
* The rest will use CPU and system RAM
* Performance may be unacceptably slow for you (maybe like 4 tokens/sec or so, but I haven’t tested)
With two GPUs:
* Most of the layers will run on the GPUs
* Inference will be a lot faster than with single GPU
* Depending on the quantization of the specific model file you download, a small portion of the model may still be run from system RAM.
Not only is the above option significantly cheaper than any Mac with 64GB+ of RAM, but with two 3090s it’s also way faster.
Edited to add: I recognize that I’m comparing the price of a new Mac to the price of used PC equipment, which might bother some. But in my experience, higher end Macs with 64GB or more hold their value extremely well, so I’m assuming for purposes of this comparison that a $4000 (new) Mac is going to be no more than 10-20% cheaper used. If you can find a high end Studio/MBP/etc for a lot less than this, it might be worth considering.
Like it or not, 96/128/192GB macs are the most affordable in VRAM/dollar.
> What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second)
That is the question I was answering. I have personally seen Llama3 70B, Q4, running via Ollama on a dual 3090 setup, and it runs about as fast as ChatGPT 4 does, and does not use main memory as far as I can tell. Yes, it's a quantized model, but the quality is pretty good, and the OP didn't specify that they must use 8 bit or greater so I figured they were going for cheap, like they said.
Edit: also, some of your figures are off: instantaneous power is measured in watts, not watt-hours or kWh, and if you want to be more efficient you can comfortably drop the max power of 3090s down to about 225W each and not lose much performance with LLM loads, as I mentioned above. Additionally, each 3090 only needs 4 lanes of PCIe 4.0 bandwidth (there are diminishing returns beyond that), so a motherboard with 48 (or even 16) free lanes is completely unnecessary. Again, I'm trying to answer the question of what the cheapest setup would be, not the ideal set up or the one with the most/fastest memory.
I don't follow the space enough to know how much work is going into llama.cpp, but from my naive outsider view, it sounds like llamafile is the place to be for "mere mortal" LLM usage. That's where they (ok, mostly she) care about those sorts of use cases.