> - 96GB of weights. You won't be able to run this on your home GPU.
You can these days, even in a portable device running on battery.
96GB fits comfortably in some laptop GPUs released this year.
You can these days, even in a portable device running on battery.
96GB fits comfortably in some laptop GPUs released this year.
Which is more than a price of RTX A6000 48gb ($4k used on ebay)
it's not very good, at all, but now we can claim some pretty massive speedups.
I can't find anything for llama 2 70B on 4090 after 10 minutes of poking around, 13B is about 30 tkn/s. it looks like people generally don't run 70B unless they have multiple 4090s.
Honestly its crazy that AMD indulges in this, especially now. Their workstation market share is comparatively tiny, and instead they could have a swarm of devs (like me) pecking away at AMD compatibility on AI repos if they sold cheap 32GB/48GB cards.
2B for the attention head and 5B from each of 2 experts.
It should be able to run slightly faster than a 13B desnse model, in as little as 16GB of RAM with room to spare.
I don't think that's the case, for full speed you still need (5B*8)/2+2~fewB overhead.
I think the experts chosen per-token? That means that yes you technically only need two in VRAM memory+router/overhead per token, but you'll have to constantly be loading in different experts unless you can fit them all, which would still be terrible for performance.
So you'll still be PCIE/RAM speed limited unless you can fit all of the experts into memory (or get really lucky and only need two experts).
Other sibling commenter refulgentis is correct too. The Apple M{1-3} Max chips have up to 400GB/s memory bandwidth. I think that's noticably faster than every other consumer CPU out there. But it's slower than a top Nvidia GPU. If the entire 96GB model has to be read by the GPU for each token, that will limit unquantised performance to 4 tokens/s at best. However, as the "Mixtral" model under discussion is a mixture-of-experts, it doesn't have to read the whole model for each token, so it might go faster. Perhaps still single-digit tokens/s though, for unquantised.