No public statement from Mistral yet. What we know:
- Mixture of Experts architecture.
- 8x 7B parameters experts (potentially trained starting with their base 7B model?).
- 96GB of weights. You won't be able to run this on your home GPU.
- Mixture of Experts architecture.
- 8x 7B parameters experts (potentially trained starting with their base 7B model?).
- 96GB of weights. You won't be able to run this on your home GPU.
People are running models 2-4 times that size on local GPUs.
What's more, this will run on a MacBook CPU just fine-- and at an extremely high speed.
This is just about right for 24GB. I bet that is intentional on their part.
This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shouldn't be catastrophic for performance in the way that being unable to fit all layers of an LLM is. It's also easy to imagine caching the N most recent experts in VRAM, where N is the largest number that still fits into your VRAM.
Even if you can't fit all of them in the VRAM, you could load everything in tmpfs, which at least removes disk I/O penalty.
In other words, assuming you ask a coding question and there's a coding expert in the mix, it would answer it completely.
If MOE models do well it could be great for commodity hw based distributed inference approaches.
You can these days, even in a portable device running on battery.
96GB fits comfortably in some laptop GPUs released this year.
Other sibling commenter refulgentis is correct too. The Apple M{1-3} Max chips have up to 400GB/s memory bandwidth. I think that's noticably faster than every other consumer CPU out there. But it's slower than a top Nvidia GPU. If the entire 96GB model has to be read by the GPU for each token, that will limit unquantised performance to 4 tokens/s at best. However, as the "Mixtral" model under discussion is a mixture-of-experts, it doesn't have to read the whole model for each token, so it might go faster. Perhaps still single-digit tokens/s though, for unquantised.
Which is more than a price of RTX A6000 48gb ($4k used on ebay)
it's not very good, at all, but now we can claim some pretty massive speedups.
I can't find anything for llama 2 70B on 4090 after 10 minutes of poking around, 13B is about 30 tkn/s. it looks like people generally don't run 70B unless they have multiple 4090s.
Honestly its crazy that AMD indulges in this, especially now. Their workstation market share is comparatively tiny, and instead they could have a swarm of devs (like me) pecking away at AMD compatibility on AI repos if they sold cheap 32GB/48GB cards.
2B for the attention head and 5B from each of 2 experts.
It should be able to run slightly faster than a 13B desnse model, in as little as 16GB of RAM with room to spare.
I don't think that's the case, for full speed you still need (5B*8)/2+2~fewB overhead.
I think the experts chosen per-token? That means that yes you technically only need two in VRAM memory+router/overhead per token, but you'll have to constantly be loading in different experts unless you can fit them all, which would still be terrible for performance.
So you'll still be PCIE/RAM speed limited unless you can fit all of the experts into memory (or get really lucky and only need two experts).
Would this allow you to run each expert on a cheap commodity GPU card so that instead of using expensive 200GB cards we can use a computer with 8 cheap gaming cards in it?
I would think no differently than you can run a large regular model on a multiGPU setup (which people do!). Its still all one network even if not all of it is activated for each token, and since its much smaller than a 56B model, it seems like there are significant components of the network that are shared.
As far as I understand in a MOE model only one/few experts are actually used at the same time, shouldn't the inference speed for this new MOE model be roughly the same as for a normal Mistral 7B then?
7B models have a reasonable throughput when ran on a beefy CPU, especially when quantized down to 4bit precision, so couldn't Mixtral be comfortably ran on a CPU too then, just with 8 times the memory footprint?
So you need roughly two loaded in memory per token. Roughly the speed and memory of a 13B per token.
Only issues is that's per-token. 2 experts are choosen per token, which means if they aren't the same ones as the last token, you need to load them into memory.
So yeah to not be disk limited you'd need roughly 8 times the memory and it would run at the speed of a 13B model.
~~~Note on quantization, iirc smaller models lose more performance when quantized vs larger models. So this would be the speed of a 4bit 13B model but with the penalty from a 4bit 7B model.~~~ Actually I have zero idea how quantization scales for MoE, I imagine it has the penalty I mentioned but that's pure speculation.
How are AMD/Intel totally missing this boat?